From patchwork Thu Mar 24 04:58:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B5EAC433EF for ; Thu, 24 Mar 2022 04:58:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345792AbiCXFAH (ORCPT ); Thu, 24 Mar 2022 01:00:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343621AbiCXFAE (ORCPT ); Thu, 24 Mar 2022 01:00:04 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32113286D0 for ; Wed, 23 Mar 2022 21:58:32 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id bi13-20020a05600c3d8d00b0038c2c33d8f3so6530930wmb.4 for ; Wed, 23 Mar 2022 21:58:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ND2tpn5Fmoo/wXsuMPeg96graaVqxq9HHzoClph6jpc=; b=pZS+ysJ2lxJWermu3BqhWniks3navL4V6ZUsqQ+iRnRC/W+8Vpa9zNTS/XCn3i4q/r QQ1Ug46PEYPIDChdHKPznaqR24jII5M3ZQd41qm5E4c8j55+fuUjGB7nRRXm1XQJKXRz +YNrYqwx409MbK53ABB0GJ1yq7nty3z7+UOWD9k1TDInklP0P48q4jOYgwByGJN01oBI TW3OiYER6diCdf4MO1iPg1apiyBZ19Zf93Ay/LsaqvVD3piefg8TXexIuxybJfxErfXK qd9He4+DFxUd/fb0HR2Ma1a3JbLKefr0wXBNAu6jgMsHBkTooSs/VZTKWlRLVlNlNoX3 K06A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ND2tpn5Fmoo/wXsuMPeg96graaVqxq9HHzoClph6jpc=; b=L2lO1o8JZTe0ohmi/Rd9FsQPliaIK5w0xR/GK+t7NWtwKvW6d+GKjjwoImNzZpN6C3 G0AEwHzT6ZgXh7XdBtPi5bFHQvYygP1dS7bw1Xxt9hWAqGgMlXEdXQwoNo1V6SKMuR8s rCzr9r3/KS9GWywM9001K2jZXF3PUOZHL6kL4M8pYbvoh9yQ56kwwIBRLqVm6bUi6+t6 s8jJCQLHM5nXIHSBH3bTqCr7KJZvFQLPXlnVJwFNOMJQcpbFdqWCZtVWienWVhKgotkM 8QrXp16MbuaVQ/jHnuSPscem8WTt2ET/ubDvuY3siFGVIQfIvUx6OZMqDuZ7RZOLsyFu UvBg== X-Gm-Message-State: AOAM530XLz1OvtTVbObJIHtjtMnp8D9EnIHmLlDIwZQh25cfIT76Ut5M 9QnIdbuALRL2koEXNLnUC9KWVh/L1Xg= X-Google-Smtp-Source: ABdhPJzwzBixEjJMreuSFwYPbOqEL7yYaG7YS7rF55IUwBk2jkCYDQ4DSlB+l1y2Ix4dIaUWQxf3SA== X-Received: by 2002:a7b:c541:0:b0:38c:b0ed:31c4 with SMTP id j1-20020a7bc541000000b0038cb0ed31c4mr12563845wmk.141.1648097910555; Wed, 23 Mar 2022 21:58:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y15-20020a056000168f00b002057a9f9f5csm1776985wrd.31.2022.03.23.21.58.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:29 -0700 (PDT) Message-Id: <53261f0099d53524155464fe79d10f9605fe93aa.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:16 +0000 Subject: [PATCH v3 01/11] bulk-checkin: rebrand plug/unplug APIs as 'odb transactions' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Make it clearer in the naming and documentation of the plug_bulk_checkin and unplug_bulk_checkin APIs that they can be thought of as a "transaction" to optimize operations on the object database. Signed-off-by: Neeraj Singh --- builtin/add.c | 4 ++-- bulk-checkin.c | 4 ++-- bulk-checkin.h | 14 ++++++++++++-- 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index 3ffb86a4338..9bf37ceae8e 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -670,7 +670,7 @@ int cmd_add(int argc, const char **argv, const char *prefix) string_list_clear(&only_match_skip_worktree, 0); } - plug_bulk_checkin(); + begin_odb_transaction(); if (add_renormalize) exit_status |= renormalize_tracked_files(&pathspec, flags); @@ -682,7 +682,7 @@ int cmd_add(int argc, const char **argv, const char *prefix) if (chmod_arg && pathspec.nr) exit_status |= chmod_pathspec(&pathspec, chmod_arg[0], show_only); - unplug_bulk_checkin(); + end_odb_transaction(); finish: if (write_locked_index(&the_index, &lock_file, diff --git a/bulk-checkin.c b/bulk-checkin.c index 6d6c37171c9..a16ae3c629d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -285,12 +285,12 @@ int index_bulk_checkin(struct object_id *oid, return status; } -void plug_bulk_checkin(void) +void begin_odb_transaction(void) { state.plugged = 1; } -void unplug_bulk_checkin(void) +void end_odb_transaction(void) { state.plugged = 0; if (state.f) diff --git a/bulk-checkin.h b/bulk-checkin.h index b26f3dc3b74..69a94422ac7 100644 --- a/bulk-checkin.h +++ b/bulk-checkin.h @@ -10,7 +10,17 @@ int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags); -void plug_bulk_checkin(void); -void unplug_bulk_checkin(void); +/* + * Tell the object database to optimize for adding + * multiple objects. end_odb_transaction must be called + * to make new objects visible. + */ +void begin_odb_transaction(void); + +/* + * Tell the object database to make any objects from the + * current transaction visible. + */ +void end_odb_transaction(void); #endif From patchwork Thu Mar 24 04:58:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31FDCC433EF for ; Thu, 24 Mar 2022 04:58:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344656AbiCXFAL (ORCPT ); Thu, 24 Mar 2022 01:00:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344514AbiCXFAF (ORCPT ); Thu, 24 Mar 2022 01:00:05 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFB74289A7 for ; Wed, 23 Mar 2022 21:58:33 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id m26-20020a05600c3b1a00b0038c8b999f58so6546031wms.1 for ; Wed, 23 Mar 2022 21:58:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=A0izl3DXxNO8poqc20QhPK4C1b+CBCwBTxzGAAuXuC4=; b=AEgZLtbuiE0CJYPThTjPkeHAk1AuqcKeiRJeiKWmLWZ+LUMKvQuAw8LVoRipQluPZ3 7hWcgNdiOID3lcDUp6N7GYgIDwP/XlEpv0ZVJpBPLcu1K5YNNaKyGN4Q8sPczxiJQz8o OnkU2zfOiaMH2s8X18pYTOHgOHqzbA4TCK11ge9gRfKJC+1URSJPURZ5Iv0yoMbMGIQ7 Hu40x8bJeysS9xSmZ/dvYxJLNiK8q4cJtBoET30dAcTJfSHl6s4PnZxW7qWkGJrzMiEi Xc8mpSaNcj+qJt3yG9KQ3A+hQRX0YH0kDGGAvsAdOn5xwLGxJUEOx6XXWPx2YpDvelpP NFdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=A0izl3DXxNO8poqc20QhPK4C1b+CBCwBTxzGAAuXuC4=; b=fzBn5Ooq5mWHXecBDvTWQy56XUpeY4w6G90+LuPdRJQ7ycAWGjqvX41bGKXO+nwPap N0pLNyIshneAvOrNMcXP2WMSB69htlCFf6Jxk0YWu+SLHZ8z5TAiNAiPBEVV7h6K+Ux4 hVq1Wi0B1ZJcq3NfxK2GjOilHKVSzJRJRciNoMXdFN9bI4N9RIuDR9GqdM7ZnZJ+Q3/x HhUeMA1gInDz7bp1Rj8twrI4waexNeh1anV77MhXLh/eakYgUI3eEv8FXCFF5z2t/X4N MAKMYKt9LYqXPH3YwjcK00jqKiO36k5qjpZnhE9Ik8TN/62+4LnmK8ylY14PD1BjqaL4 lGug== X-Gm-Message-State: AOAM531gJAxVVCAdAw3oIp/3U85wfQdMzXQhz6giQQkKEjGibcHA4hFS Eu4OEVdZxDooj+r/mlO767o2e+FqtNw= X-Google-Smtp-Source: ABdhPJx/b55Z2XaEo/V29e7YE9kofSWt851oolQUi32KNYwq0i8IMG4wLPJLHyA9n4UG4CO8RCSMCg== X-Received: by 2002:a05:600c:1c11:b0:38b:4924:9a77 with SMTP id j17-20020a05600c1c1100b0038b49249a77mr421814wms.0.1648097912282; Wed, 23 Mar 2022 21:58:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n8-20020a5d5988000000b00203d5f1f3e4sm1576905wri.105.2022.03.23.21.58.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:31 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:17 +0000 Subject: [PATCH v3 02/11] bulk-checkin: rename 'state' variable and separate 'plugged' boolean Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit prepares for adding batch-fsync to the bulk-checkin infrastructure. The bulk-checkin infrastructure is currently used to batch up addition of large blobs to a packfile. When a blob is larger than big_file_threshold, we unconditionally add it to a pack. If bulk checkins are 'plugged', we allow multiple large blobs to be added to a single pack until we reach the packfile size limit; otherwise, we simply make a new packfile for each large blob. The 'unplug' call tells us when the series of blob additions is done so that we can finish the packfiles and make their objects available to subsequent operations. Stated another way, bulk-checkin allows callers to define a transaction that adds multiple objects to the object database, where the object database can optimize its internal operations within the transaction boundary. Batched fsync will fit into bulk-checkin by taking advantage of the plug/unplug functionality to determine the appropriate time to fsync and make newly-added objects available in the primary object database. * Rename 'state' variable to 'bulk_checkin_state', since we will later be adding 'bulk_fsync_objdir'. This also makes the variable easier to find in the debugger, since the name is more unique. * Move the 'plugged' data member of 'bulk_checkin_state' into a separate static variable. Doing this avoids resetting the variable in finish_bulk_checkin when zeroing the 'bulk_checkin_state'. As-is, we seem to unintentionally disable the plugging functionality the first time a new packfile must be created due to packfile size limits. While disabling the plugging state only results in suboptimal behavior for the current code, it would be fatal for the bulk-fsync functionality later in this patch series. Signed-off-by: Neeraj Singh --- bulk-checkin.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/bulk-checkin.c b/bulk-checkin.c index a16ae3c629d..ffe142841b2 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -10,9 +10,9 @@ #include "packfile.h" #include "object-store.h" -static struct bulk_checkin_state { - unsigned plugged:1; +static int bulk_checkin_plugged; +static struct bulk_checkin_state { char *pack_tmp_name; struct hashfile *f; off_t offset; @@ -21,7 +21,7 @@ static struct bulk_checkin_state { struct pack_idx_entry **written; uint32_t alloc_written; uint32_t nr_written; -} state; +} bulk_checkin_state; static void finish_tmp_packfile(struct strbuf *basename, const char *pack_tmp_name, @@ -278,21 +278,23 @@ int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags) { - int status = deflate_to_pack(&state, oid, fd, size, type, + int status = deflate_to_pack(&bulk_checkin_state, oid, fd, size, type, path, flags); - if (!state.plugged) - finish_bulk_checkin(&state); + if (!bulk_checkin_plugged) + finish_bulk_checkin(&bulk_checkin_state); return status; } void begin_odb_transaction(void) { - state.plugged = 1; + assert(!bulk_checkin_plugged); + bulk_checkin_plugged = 1; } void end_odb_transaction(void) { - state.plugged = 0; - if (state.f) - finish_bulk_checkin(&state); + assert(bulk_checkin_plugged); + bulk_checkin_plugged = 0; + if (bulk_checkin_state.f) + finish_bulk_checkin(&bulk_checkin_state); } From patchwork Thu Mar 24 04:58:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61C74C433F5 for ; Thu, 24 Mar 2022 04:58:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346146AbiCXFAP (ORCPT ); Thu, 24 Mar 2022 01:00:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345226AbiCXFAG (ORCPT ); Thu, 24 Mar 2022 01:00:06 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D90C828E35 for ; Wed, 23 Mar 2022 21:58:34 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id 123-20020a1c1981000000b0038b3616a71aso1935569wmz.4 for ; Wed, 23 Mar 2022 21:58:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=W9Vimte0sGyR2DMe0rSIsRe6vitxNiJv2vsesuc0ht0=; b=fvNZ6wDZOqUoDYoSz8bTia4MCTgo9ySe9mCp3ZLZIzqmPCb+lM2L+7KNQtyzZXG1Ev cuL0biuTH6mb8nPt5jlMzbWdQmwgZ1guLbDXCsm1v0oCdc15TQYRk0g9sBLbzAjbETlF I1aRS0bNf+PSCzy2JbLai289wWq7LuFZjF+GAv+lWpLdnPOxYIyhCWdNnoyIEVXwdLaM 5m4owifR/TFyeub95jVqSn3zsLKjYzfIZZ7OntMHxMSZ2cuyy+5c28OlsqAsQZm5U132 JpxlQc5BObX7GkwEg+Tw8M8lFPpAqOTzcOng4UfOXDBoTSk+7vlJTCoB7kJaePMuL8Ez n3+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=W9Vimte0sGyR2DMe0rSIsRe6vitxNiJv2vsesuc0ht0=; b=v4WsKi+edcp79dbnc4OtTbtX05fRT7JKHfdpA0PsVSLDNDa8RmC98kjnDgbC4HfhIu E486ddGVgFJh3MDs1maW+yY1i9GAgVGXKQB0kJd0lD8dhqj7BLaIMUUHcCZuiNirZPez 2Ziy03TUL77zuPakmmJvzyOtDouQnJkLeD2RD7gXeEcANmokIVct8yLg2ntIH/nwIn0J 4JGp+91lD95s1UMJhR4XszZxPtq9UGLpDw+FJtOrKJD5D0YX0ldWRapOFcnrILhkJtAb 4W8dsy7JBMFqhES7FBTjxmXRlaqVqsjUSGcyLf7t0QzV1TB+SJSgtR25+4CE14/WT503 E32Q== X-Gm-Message-State: AOAM533aYQ+xt+SsiqKEY0CTFqTCIzOz8DUXR8pDWYPk9OofkFBzJRBM /s9Rj6yjAsMo4a5HgCMElvIKAIDWtRc= X-Google-Smtp-Source: ABdhPJzo5SkGO3MZM4JD/SKqSWuQtDWnuRNG+SxOAX99TnYp3cgKzXOxHe4jBJNpoTqOAqySN0gMGA== X-Received: by 2002:a05:600c:4296:b0:38c:1b43:1562 with SMTP id v22-20020a05600c429600b0038c1b431562mr2993397wmc.122.1648097913331; Wed, 23 Mar 2022 21:58:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l15-20020a05600c4f0f00b0038cbdf5221dsm4176349wmq.41.2022.03.23.21.58.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:32 -0700 (PDT) Message-Id: <26ce5b8fddaa507e09d3201c4c6492e4d1184d53.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:18 +0000 Subject: [PATCH v3 03/11] object-file: pass filename to fsync_or_die Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh If we die while trying to fsync a loose object file, pass the actual filename we're trying to sync. This is likely to be more helpful for a user trying to diagnose the cause of the failure than the former 'loose object file' string. It also sidesteps any concerns about translating the die message differently for loose objects versus something else that has a real path. Signed-off-by: Neeraj Singh --- object-file.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/object-file.c b/object-file.c index b254bc50d70..5ffbf3d4fd4 100644 --- a/object-file.c +++ b/object-file.c @@ -1888,16 +1888,16 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf, } /* Finalize a file on disk, and close it. */ -static void close_loose_object(int fd) +static void close_loose_object(int fd, const char *filename) { if (the_repository->objects->odb->will_destroy) goto out; if (fsync_object_files > 0) - fsync_or_die(fd, "loose object file"); + fsync_or_die(fd, filename); else fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, - "loose object file"); + filename); out: if (close(fd) != 0) @@ -2011,7 +2011,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr, die(_("confused by unstable object source data for %s"), oid_to_hex(oid)); - close_loose_object(fd); + close_loose_object(fd, tmp_file.buf); if (mtime) { struct utimbuf utb; From patchwork Thu Mar 24 04:58:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22FF7C433F5 for ; Thu, 24 Mar 2022 04:58:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346355AbiCXFAQ (ORCPT ); Thu, 24 Mar 2022 01:00:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345865AbiCXFAI (ORCPT ); Thu, 24 Mar 2022 01:00:08 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32BC929C81 for ; Wed, 23 Mar 2022 21:58:36 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id m30so4965219wrb.1 for ; Wed, 23 Mar 2022 21:58:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3XKfyW8DARg+BP1BgM5+fxxF8dikwvwu5ztym6Mrapo=; b=EGItsD4+sVCXcEIQS1jt4uuytRxrrebnM93KYWO7kg28f7IbnCk9UQhFPkh8QPJc4N lwT8tRfRn+MTUz5ncGYlE21kjeznLFAXDXGSIdWDHJ2stFSrg3u1n7QhyE/8HEm6pyGd 3GlCUsjYque/iIP2nkTXgJbuNBTMt59pDrpD1u/gKXqGSDyjkgCZN26fsvbhpGcDAqv/ Ibe4i789IB+LtJk6kT8M0dGyo1mrHtTyoJCi0SZuNSNDNZ/Yfe1XFe0M0USOCRWlEoLI YE5+021iQlPI/1/9iRyvSmnwlSDyc+9Km4pJefbKztwInbkJ7D4SfrmuRNLNgH8cGmeF HQgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3XKfyW8DARg+BP1BgM5+fxxF8dikwvwu5ztym6Mrapo=; b=AQ/WtkYQNbZtpRDK1IVaT22LUUymQuR+O2kQrVkD54WRIEIIopuKRi25KECN2uj1xD oEaBpbxJAC0OMQa19vXTCIpMe9yjQgBuYRxKdFndpoWZ8MUH6Z013g5THjzO0oSo34Ng edNutdFX4Pla3nWbHG3cWV6OCxg3CqlGupI14ju22Ry706TOWDVyqvZW1NjiiIzusosh p+WWu7v9D1X4/3jjq0x9sRTpiuP5Ghz1/i1sgRNrdRyXZ++mbGhc0Ckbz5YoSGVCOdDr o1JjmpjMAPpPh3cWkyfVHE659F230GZepls0iohMKHpr5bvgcmhMqVNz+s5z7oOy2zE7 vkTw== X-Gm-Message-State: AOAM530jRs6cbKgv80A6hvQ51d26kWWnajXsF9oeTNBU8rfhFLLHGxNY G9ctXTDQv3ucoGit9hPK08RVja/6H5A= X-Google-Smtp-Source: ABdhPJz3LQhtfhSA17YKCFFAM5KpD0POsHMw8tLISXcWZmeNNwNRqKpuZpmKhE/FBHO4JSoFC8YEbw== X-Received: by 2002:a05:6000:188f:b0:205:4d98:607a with SMTP id a15-20020a056000188f00b002054d98607amr2833109wri.505.1648097914555; Wed, 23 Mar 2022 21:58:34 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g10-20020adfe40a000000b00203eb3551f0sm2134982wrm.117.2022.03.23.21.58.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:33 -0700 (PDT) Message-Id: <52638326790aaf5113ff64c74023100beb90760d.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:19 +0000 Subject: [PATCH v3 04/11] core.fsyncmethod: batched disk flushes for loose-objects Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh When adding many objects to a repo with `core.fsync=loose-object`, the cost of fsync'ing each object file can become prohibitive. One major source of the cost of fsync is the implied flush of the hardware writeback cache within the disk drive. This commit introduces a new `core.fsyncMethod=batch` option that batches up hardware flushes. It hooks into the bulk-checkin odb-transaction functionality, takes advantage of tmp-objdir, and uses the writeout-only support code. When the new mode is enabled, we do the following for each new object: 1a. Create the object in a tmp-objdir. 2a. Issue a pagecache writeback request and wait for it to complete. At the end of the entire transaction when unplugging bulk checkin: 1b. Issue an fsync against a dummy file to flush the log and hardware writeback cache, which should by now have seen the tmp-objdir writes. 2b. Rename all of the tmp-objdir files to their final names. 3b. When updating the index and/or refs, we assume that Git will issue another fsync internal to that operation. This is not the default today, but the user now has the option of syncing the index and there is a separate patch series to implement syncing of refs. On a filesystem with a singular journal that is updated during name operations (e.g. create, link, rename, etc), such as NTFS, HFS+, or XFS we would expect the fsync to trigger a journal writeout so that this sequence is enough to ensure that the user's data is durable by the time the git command returns. This sequence also ensures that no object files appear in the main object store unless they are fsync-durable. Batch mode is only enabled if core.fsync includes loose-objects. If the legacy core.fsyncObjectFiles setting is enabled, but core.fsync does not include loose-objects, we will use file-by-file fsyncing. In step (1a) of the sequence, the tmp-objdir is created lazily to avoid work if no loose objects are ever added to the ODB. We use a tmp-objdir to maintain the invariant that no loose-objects are visible in the main ODB unless they are properly fsync-durable. This is important since future ODB operations that try to create an object with specific contents will silently drop the new data if an object with the target hash exists without checking that the loose-object contents match the hash. Only a full git-fsck would restore the ODB to a functional state where dataloss doesn't occur. In step (1b) of the sequence, we issue a fsync against a dummy file created specifically for the purpose. This method has a little higher cost than using one of the input object files, but makes adding new callers of this mechanism easier, since we don't need to figure out which object file is "last" or risk sharing violations by caching the fd of the last object file. _Performance numbers_: Linux - Hyper-V VM running Kernel 5.11 (Ubuntu 20.04) on a fast SSD. Mac - macOS 11.5.1 running on a Mac mini on a 1TB Apple SSD. Windows - Same host as Linux, a preview version of Windows 11. Adding 500 files to the repo with 'git add' Times reported in seconds. object file syncing | Linux | Mac | Windows --------------------|-------|-------|-------- disabled | 0.06 | 0.35 | 0.61 fsync | 1.88 | 11.18 | 2.47 batch | 0.15 | 0.41 | 1.53 Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 8 ++++ bulk-checkin.c | 71 +++++++++++++++++++++++++++++++++++ bulk-checkin.h | 3 ++ cache.h | 8 +++- config.c | 2 + object-file.c | 7 +++- 6 files changed, 97 insertions(+), 2 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 9da3e5d88f6..3c90ba0b395 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -596,6 +596,14 @@ core.fsyncMethod:: * `writeout-only` issues pagecache writeback requests, but depending on the filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. +* `batch` enables a mode that uses writeout-only flushes to stage multiple + updates in the disk writeback cache and then does a single full fsync of + a dummy file to trigger the disk cache flush at the end of the operation. ++ + Currently `batch` mode only applies to loose-object files. Other repository + data is made durable as if `fsync` was specified. This mode is expected to + be as safe as `fsync` on macOS for repos stored on HFS+ or APFS filesystems + and on Windows for repos stored on NTFS or ReFS filesystems. core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. diff --git a/bulk-checkin.c b/bulk-checkin.c index ffe142841b2..a5c40a08b8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -3,15 +3,20 @@ */ #include "cache.h" #include "bulk-checkin.h" +#include "lockfile.h" #include "repository.h" #include "csum-file.h" #include "pack.h" #include "strbuf.h" +#include "string-list.h" +#include "tmp-objdir.h" #include "packfile.h" #include "object-store.h" static int bulk_checkin_plugged; +static struct tmp_objdir *bulk_fsync_objdir; + static struct bulk_checkin_state { char *pack_tmp_name; struct hashfile *f; @@ -80,6 +85,40 @@ clear_exit: reprepare_packed_git(the_repository); } +/* + * Cleanup after batch-mode fsync_object_files. + */ +static void do_batch_fsync(void) +{ + struct strbuf temp_path = STRBUF_INIT; + struct tempfile *temp; + + if (!bulk_fsync_objdir) + return; + + /* + * Issue a full hardware flush against a temporary file to ensure + * that all objects are durable before any renames occur. The code in + * fsync_loose_object_bulk_checkin has already issued a writeout + * request, but it has not flushed any writeback cache in the storage + * hardware or any filesystem logs. This fsync call acts as a barrier + * to ensure that the data in each new object file is durable before + * the final name is visible. + */ + strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX", get_object_directory()); + temp = xmks_tempfile(temp_path.buf); + fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp)); + delete_tempfile(&temp); + strbuf_release(&temp_path); + + /* + * Make the object files visible in the primary ODB after their data is + * fully durable. + */ + tmp_objdir_migrate(bulk_fsync_objdir); + bulk_fsync_objdir = NULL; +} + static int already_written(struct bulk_checkin_state *state, struct object_id *oid) { int i; @@ -274,6 +313,36 @@ static int deflate_to_pack(struct bulk_checkin_state *state, return 0; } +void prepare_loose_object_bulk_checkin(void) +{ + /* + * We lazily create the temporary object directory + * the first time an object might be added, since + * callers may not know whether any objects will be + * added at the time they call begin_odb_transaction. + */ + if (!bulk_checkin_plugged || bulk_fsync_objdir) + return; + + bulk_fsync_objdir = tmp_objdir_create("bulk-fsync"); + if (bulk_fsync_objdir) + tmp_objdir_replace_primary_odb(bulk_fsync_objdir, 0); +} + +void fsync_loose_object_bulk_checkin(int fd, const char *filename) +{ + /* + * If we have a plugged bulk checkin, we issue a call that + * cleans the filesystem page cache but avoids a hardware flush + * command. Later on we will issue a single hardware flush + * before as part of do_batch_fsync. + */ + if (!bulk_fsync_objdir || + git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) { + fsync_or_die(fd, filename); + } +} + int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags) @@ -297,4 +366,6 @@ void end_odb_transaction(void) bulk_checkin_plugged = 0; if (bulk_checkin_state.f) finish_bulk_checkin(&bulk_checkin_state); + + do_batch_fsync(); } diff --git a/bulk-checkin.h b/bulk-checkin.h index 69a94422ac7..70edf745be8 100644 --- a/bulk-checkin.h +++ b/bulk-checkin.h @@ -6,6 +6,9 @@ #include "cache.h" +void prepare_loose_object_bulk_checkin(void); +void fsync_loose_object_bulk_checkin(int fd, const char *filename); + int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags); diff --git a/cache.h b/cache.h index ef7d34b7a09..a5bf15a5131 100644 --- a/cache.h +++ b/cache.h @@ -1040,7 +1040,8 @@ extern int use_fsync; enum fsync_method { FSYNC_METHOD_FSYNC, - FSYNC_METHOD_WRITEOUT_ONLY + FSYNC_METHOD_WRITEOUT_ONLY, + FSYNC_METHOD_BATCH, }; extern enum fsync_method fsync_method; @@ -1767,6 +1768,11 @@ void fsync_or_die(int fd, const char *); int fsync_component(enum fsync_component component, int fd); void fsync_component_or_die(enum fsync_component component, int fd, const char *msg); +static inline int batch_fsync_enabled(enum fsync_component component) +{ + return (fsync_components & component) && (fsync_method == FSYNC_METHOD_BATCH); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/config.c b/config.c index 3c9b6b589ab..511f4584eeb 100644 --- a/config.c +++ b/config.c @@ -1688,6 +1688,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb) fsync_method = FSYNC_METHOD_FSYNC; else if (!strcmp(value, "writeout-only")) fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else if (!strcmp(value, "batch")) + fsync_method = FSYNC_METHOD_BATCH; else warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); diff --git a/object-file.c b/object-file.c index 5ffbf3d4fd4..d2e0c13198f 100644 --- a/object-file.c +++ b/object-file.c @@ -1893,7 +1893,9 @@ static void close_loose_object(int fd, const char *filename) if (the_repository->objects->odb->will_destroy) goto out; - if (fsync_object_files > 0) + if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT)) + fsync_loose_object_bulk_checkin(fd, filename); + else if (fsync_object_files > 0) fsync_or_die(fd, filename); else fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, @@ -1961,6 +1963,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr, static struct strbuf tmp_file = STRBUF_INIT; static struct strbuf filename = STRBUF_INIT; + if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT)) + prepare_loose_object_bulk_checkin(); + loose_object_path(the_repository, &filename, oid); fd = create_tmpfile(&tmp_file, filename.buf); From patchwork Thu Mar 24 04:58:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EDAEC433EF for ; Thu, 24 Mar 2022 04:58:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346463AbiCXFAV (ORCPT ); Thu, 24 Mar 2022 01:00:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345915AbiCXFAI (ORCPT ); Thu, 24 Mar 2022 01:00:08 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34D052A241 for ; Wed, 23 Mar 2022 21:58:37 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id t11so4942892wrm.5 for ; Wed, 23 Mar 2022 21:58:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=dGhuKNFF6d1Xb8n8Bmr/hlaiPKd4B21RTCn9HBUk55Y=; b=m2ahk5Ce6u7wJFWNzjBlZtPfP+5B7JEXWI6php1qxe9TYrwsj5c3VDOnUd67QBjdru A/CtWzW+5OyPNr78ZW7zf7ogox3KZV6bXwYwwqwNlwE8RqRq0dApohPfZozPo960lMSo uxH0VV9aguS+upJ179YnaRMrQ1RLQBTzDFt8vTIj3dMSFSp9TQJA1oj6eyIQYS/6WGqe favT32CwV5FXCAdiqzTivslaShFQHNVtC6Ipuw5vNtC80HDV9mQ20d8QegZgq2df4D7F TAy+c/Md9ifYMPzdgkYS/m3YfLrtxu6X0aiZvc61E41LRKkofFUQlDRJU0o7fxz8aYUz MdAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=dGhuKNFF6d1Xb8n8Bmr/hlaiPKd4B21RTCn9HBUk55Y=; b=Vu0IqaXaR81R5buKq8BK8R7vn9iB/CFMAChNe/silsA+WZys4G97YBKB000VEuZBMp mvWd5BOajJNO4NtzrSf+DWHVVmwH1bHFz/51B3XmMZ2GtBXUFyVcNmJxvqDZx+U6a21l gj1eFKkL54XoBkIWtsbMu20jGEE60fp8KipLiV/4a/tHHMoELIlwrtqowvt/qcZ3PjmA b6ZE+b+HbRcz51doHKnZKMthZm8fePHAgZpkh7J7MJM4KSPiAgWepoPrdtAlsRFT9OGn iLHbSlh8HuJ3huCboXL1gLYhS1HkTECTGk35tReZ/1sEmoyDVfdkoaDhtnVO92nZRp9h PZIQ== X-Gm-Message-State: AOAM533kuSricNi4b75yprg+0e+g5DRTHhXHCY4cw+38Aq+7hmVztSeb 7zLt0nKe4yBwHWzjgvQhXMriKKlECos= X-Google-Smtp-Source: ABdhPJxGNDR1GqF5DgBhaPberFxCs/Evii/BzH+7YIctUytgQbpe7D6ZFJQuHApR7epbfFI1KVcaxw== X-Received: by 2002:a5d:6292:0:b0:203:d745:5a94 with SMTP id k18-20020a5d6292000000b00203d7455a94mr2903958wru.286.1648097915639; Wed, 23 Mar 2022 21:58:35 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m23-20020a05600c3b1700b0038bbd24f401sm4868557wms.2.2022.03.23.21.58.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:35 -0700 (PDT) Message-Id: <913ce1b3df9cf273f1572c256dffad1cacc192a6.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:20 +0000 Subject: [PATCH v3 05/11] update-index: use the bulk-checkin infrastructure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The update-index functionality is used internally by 'git stash push' to setup the internal stashed commit. This change enables odb-transactions for update-index infrastructure to speed up adding new objects to the object database by leveraging the batch fsync functionality. There is some risk with this change, since under batch fsync, the object files will be in a tmp-objdir until update-index is complete, so callers using the --stdin option will not see them until update-index is done. This risk is mitigated by unplugging the batch when reporting verbose output, which is the only way a --stdin caller might synchronize with the addition of an object. Signed-off-by: Neeraj Singh --- builtin/update-index.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/builtin/update-index.c b/builtin/update-index.c index aafe7eeac2a..ae7887cfe37 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -5,6 +5,7 @@ */ #define USE_THE_INDEX_COMPATIBILITY_MACROS #include "cache.h" +#include "bulk-checkin.h" #include "config.h" #include "lockfile.h" #include "quote.h" @@ -32,6 +33,7 @@ static int allow_replace; static int info_only; static int force_remove; static int verbose; +static int odb_transaction_active; static int mark_valid_only; static int mark_skip_worktree_only; static int mark_fsmonitor_only; @@ -49,6 +51,15 @@ enum uc_mode { UC_FORCE }; +static void end_odb_transaction_if_active(void) +{ + if (!odb_transaction_active) + return; + + end_odb_transaction(); + odb_transaction_active = 0; +} + __attribute__((format (printf, 1, 2))) static void report(const char *fmt, ...) { @@ -57,6 +68,16 @@ static void report(const char *fmt, ...) if (!verbose) return; + /* + * It is possible, though unlikely, that a caller + * could use the verbose output to synchronize with + * addition of objects to the object database, so + * unplug bulk checkin to make sure that future objects + * are immediately visible. + */ + + end_odb_transaction_if_active(); + va_start(vp, fmt); vprintf(fmt, vp); putchar('\n'); @@ -1116,6 +1137,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) */ parse_options_start(&ctx, argc, argv, prefix, options, PARSE_OPT_STOP_AT_NON_OPTION); + + /* + * Allow the object layer to optimize adding multiple objects in + * a batch. + */ + begin_odb_transaction(); + odb_transaction_active = 1; while (ctx.argc) { if (parseopt_state != PARSE_OPT_DONE) parseopt_state = parse_options_step(&ctx, options, @@ -1190,6 +1218,11 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) strbuf_release(&buf); } + /* + * By now we have added all of the new objects + */ + end_odb_transaction_if_active(); + if (split_index > 0) { if (git_config_get_split_index() == 0) warning(_("core.splitIndex is set to false; " From patchwork Thu Mar 24 04:58:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0A6DC433FE for ; Thu, 24 Mar 2022 04:58:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346491AbiCXFAW (ORCPT ); Thu, 24 Mar 2022 01:00:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343621AbiCXFAK (ORCPT ); Thu, 24 Mar 2022 01:00:10 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9115C2A715 for ; Wed, 23 Mar 2022 21:58:38 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id v130-20020a1cac88000000b00389d0a5c511so6512651wme.5 for ; Wed, 23 Mar 2022 21:58:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5ebJFCI4F0cWCTTRJ8U340O4g9iVrZ+6k6F3XALLE/c=; b=jZPpODWadMEqA6kN+VmIWir2miYSpIScwNR2tDRUH1ZcAhR4s4LzCR7btq6sHpLBDr Qhl7ZmyCZZweZsQOTYXCg5S1orA+WrM8Ni9hngPWZYnKxwzHDBeXsxIOLbahqYAcqB5A /cixAlTuekhXPV44QgDWieCJGAtoukx4DbnwmDRiijGCCTxREG6+UGhH+S37ryCzrdLX IGGXsnHZtdx2Z7PWwcw1qCTJFSvXtGO8ErcCc+NvSd7aTmojQEu112ks7ftUT366OyY6 6FSCoLxUKSHz/HhCk+xPIEi3N27KaFThih5AhxLzvpRLnGJ9vWpP6dhf0oMnusfFUg8z iDjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5ebJFCI4F0cWCTTRJ8U340O4g9iVrZ+6k6F3XALLE/c=; b=wCL5LkAk9xkpsNWnUaaBzJ64UXTf2SNNINkcCHhlUOdPZkbwawhuiaA8SCd011bQ0O dbb1L7qTebkGr7o8WHGIYakUY2aTMrcZmvDyNB4GJz/zcVuBQMkZXJOl9YRRsu/RX5Wu AIUTdAaLJQDO3h949s2rDjEr4GaRTNPEu0qJu2lw7KZF4oS9s49izZnUcTx2zjCn3Y9l 4tzlAmfMNQ6wFFDvxvpcH653VGl7ltoIc/rBUIDASACNvAW74wz6Wx7vsLfQ50SwhRQn 78TfUTMcz/QDs/qEJaQYClgU3YF03w7pCV5cgg5v2CPJk9ueoO8HioBfb9NN8iXeJeZy XQeQ== X-Gm-Message-State: AOAM533BwrAgCOHonoYL27ZMwV21STwo75YtRQarb/YlmbvLxXGhhyqE g4jdfBiVs5aayEZc4u6KfgzAQUybHxA= X-Google-Smtp-Source: ABdhPJy2QmpgfmNTeyPq4fAHN7koOu9cvWk9UWQFBHWtw08BIfdAMjj1bKEqX5+RgiUbzDMrCUJ4MA== X-Received: by 2002:a05:600c:4f15:b0:38c:b729:4838 with SMTP id l21-20020a05600c4f1500b0038cb7294838mr2986582wmq.132.1648097917083; Wed, 23 Mar 2022 21:58:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c5-20020a5d63c5000000b002040822b680sm2300831wrw.81.2022.03.23.21.58.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:36 -0700 (PDT) Message-Id: <84fd144ef1889aaf4f88040a60cb8b156522e1b4.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:21 +0000 Subject: [PATCH v3 06/11] unpack-objects: use the bulk-checkin infrastructure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The unpack-objects functionality is used by fetch, push, and fast-import to turn the transfered data into object database entries when there are fewer objects than the 'unpacklimit' setting. By enabling an odb-transaction when unpacking objects, we can take advantage of batched fsyncs. Here are some performance numbers to justify batch mode for unpack-objects, collected on a WSL2 Ubuntu VM. Fsync Mode | Time for 90 objects (ms) ------------------------------------- Off | 170 On,fsync | 760 On,batch | 230 Note that the default unpackLimit is 100 objects, so there's a 3x benefit in the worst case. The non-batch mode fsync scales linearly with the number of objects, so there are significant benefits even with smaller numbers of objects. Signed-off-by: Neeraj Singh --- builtin/unpack-objects.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c index dbeb0680a58..56d05e2725d 100644 --- a/builtin/unpack-objects.c +++ b/builtin/unpack-objects.c @@ -1,5 +1,6 @@ #include "builtin.h" #include "cache.h" +#include "bulk-checkin.h" #include "config.h" #include "object-store.h" #include "object.h" @@ -503,10 +504,12 @@ static void unpack_all(void) if (!quiet) progress = start_progress(_("Unpacking objects"), nr_objects); CALLOC_ARRAY(obj_list, nr_objects); + begin_odb_transaction(); for (i = 0; i < nr_objects; i++) { unpack_one(i); display_progress(progress, i + 1); } + end_odb_transaction(); stop_progress(&progress); if (delta_list) From patchwork Thu Mar 24 04:58:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90310C433EF for ; Thu, 24 Mar 2022 04:59:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347009AbiCXFAb (ORCPT ); Thu, 24 Mar 2022 01:00:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345848AbiCXFAM (ORCPT ); Thu, 24 Mar 2022 01:00:12 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F23A02AC68 for ; Wed, 23 Mar 2022 21:58:40 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id bg31-20020a05600c3c9f00b00381590dbb33so1940771wmb.3 for ; Wed, 23 Mar 2022 21:58:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=hg8plsugjefXxhMhSvzABM0c1aAu2C8TxytBC4+PwWI=; b=ElIyKQkVTvLZAPH30rX17lO4UK/VA0bS8CH8nyrkFiqmrWUEPZTXQyqdk7WEfCF+o8 j7Eodr1wM3e65Kljzji9mApGh15I+JBth2ekSw1TxcFTN9tFJagNCZas1evoc+wuVISL JKZzEc2RiYiTeZe4B0CxVQbk5HBa1I9YVaEzfVcNY1asqb6YE/N5NXSMibZP2nvoyuJg vNEdBHDjJmJa57Y4Jl472UMknptG+oor4g/FxRH+X6dHkFTQkwpP7Uj50/ISVVneLLX1 I+vTWzBxefN+Fhf+6hYsq6Wi4ULXeEsHjgbb4WN1KK3izX9GM2RMMqVgtc7aLbh+G0St lbQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=hg8plsugjefXxhMhSvzABM0c1aAu2C8TxytBC4+PwWI=; b=S7Qv3AA11xhTLQK2hg/Vk+dQ/FYLbVepi4KPBdgR+4rvxj9GY+xQbvPvgLL0sodBgp 2Grk9ZXmH3qZGd8fQQlVzsdf1qGEGHTiVPdsChsc1tLuqAgSe8yuWHXG9LoQ5Bs48v6D eCfvBrGFGwnvMOGcYObGDlK1VUwlkaatPyBTqwKATaqzpQiGUWy9Y3OHDOLMnDZfwwz5 HEB4gs+qq3aZbaw6WtOsSaWJvjzUuSSub9c4EoctIpJDsy8ODL+SPzBCBaY89xBqaYUY kgo6jvmFOPrMXEtrSTPXgW6sbTEnMygbZk8p7ttrPnvM1gdRsM+Nl6zfQeHXNF7tzF0V JC7w== X-Gm-Message-State: AOAM530Eeq/XH3olR2DmulqCOjbJz4tiL3e2iGHknWjpsLKNphVlnUUO MM1oC2skPj99jE+PDRVG/du3Iq11ccs= X-Google-Smtp-Source: ABdhPJx6Dx0egO6gInE/n0JpIi+sLd5yGzI+4vDE1XcdGCq6Jn6HdoP8AhTfjd4QPXc8mkI30m2xdw== X-Received: by 2002:a05:600c:3487:b0:38c:9a42:4d49 with SMTP id a7-20020a05600c348700b0038c9a424d49mr3067487wmq.29.1648097919074; Wed, 23 Mar 2022 21:58:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o4-20020a5d6484000000b002057ad822d4sm1719918wri.48.2022.03.23.21.58.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:38 -0700 (PDT) Message-Id: <447263e8ef14021714976e489248cdcc48ec0303.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:22 +0000 Subject: [PATCH v3 07/11] core.fsync: use batch mode and sync loose objects by default on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Git for Windows has defaulted to core.fsyncObjectFiles=true since September 2017. We turn on syncing of loose object files with batch mode in upstream Git so that we can get broad coverage of the new code upstream. We don't actually do fsyncs in the most of the test suite, since GIT_TEST_FSYNC is set to 0. However, we do exercise all of the surrounding batch mode code since GIT_TEST_FSYNC merely makes the maybe_fsync wrapper always appear to succeed. Signed-off-by: Neeraj Singh --- cache.h | 4 ++++ compat/mingw.h | 3 +++ config.c | 2 +- git-compat-util.h | 2 ++ 4 files changed, 10 insertions(+), 1 deletion(-) diff --git a/cache.h b/cache.h index a5bf15a5131..7f6cbb254b4 100644 --- a/cache.h +++ b/cache.h @@ -1031,6 +1031,10 @@ enum fsync_component { FSYNC_COMPONENT_INDEX | \ FSYNC_COMPONENT_REFERENCE) +#ifndef FSYNC_COMPONENTS_PLATFORM_DEFAULT +#define FSYNC_COMPONENTS_PLATFORM_DEFAULT FSYNC_COMPONENTS_DEFAULT +#endif + /* * A bitmask indicating which components of the repo should be fsynced. */ diff --git a/compat/mingw.h b/compat/mingw.h index 6074a3d3ced..afe30868c04 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -332,6 +332,9 @@ int mingw_getpagesize(void); int win32_fsync_no_flush(int fd); #define fsync_no_flush win32_fsync_no_flush +#define FSYNC_COMPONENTS_PLATFORM_DEFAULT (FSYNC_COMPONENTS_DEFAULT | FSYNC_COMPONENT_LOOSE_OBJECT) +#define FSYNC_METHOD_DEFAULT (FSYNC_METHOD_BATCH) + struct rlimit { unsigned int rlim_cur; }; diff --git a/config.c b/config.c index 511f4584eeb..e9cac5f4707 100644 --- a/config.c +++ b/config.c @@ -1342,7 +1342,7 @@ static const struct fsync_component_name { static enum fsync_component parse_fsync_components(const char *var, const char *string) { - enum fsync_component current = FSYNC_COMPONENTS_DEFAULT; + enum fsync_component current = FSYNC_COMPONENTS_PLATFORM_DEFAULT; enum fsync_component positive = 0, negative = 0; while (string) { diff --git a/git-compat-util.h b/git-compat-util.h index 0892e209a2f..fffe42ce7c1 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1257,11 +1257,13 @@ __attribute__((format (printf, 3, 4))) NORETURN void BUG_fl(const char *file, int line, const char *fmt, ...); #define BUG(...) BUG_fl(__FILE__, __LINE__, __VA_ARGS__) +#ifndef FSYNC_METHOD_DEFAULT #ifdef __APPLE__ #define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY #else #define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC #endif +#endif enum fsync_action { FSYNC_WRITEOUT_ONLY, From patchwork Thu Mar 24 04:58:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9EDEC433F5 for ; Thu, 24 Mar 2022 04:59:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346180AbiCXFAp (ORCPT ); Thu, 24 Mar 2022 01:00:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346265AbiCXFAQ (ORCPT ); Thu, 24 Mar 2022 01:00:16 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC56B2B18D for ; Wed, 23 Mar 2022 21:58:41 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id r13so4925102wrr.9 for ; Wed, 23 Mar 2022 21:58:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=f/ZbN5quD6ggV2+4j9lRbjM6zktg3vb1d35PK4/3tT4=; b=V+DDuV+a+prxUkTSHvXDayTX7LEsI92hd/ZeK/3jock+iJNZbdJjCdmpG4kZetD49E 0DqxAgFct2PQc8RJ0qm12FxY3Nds0rO0VM/eTwPiibx0UZ3F7tSlyBrLt3d7XFDGfrXW wlR8g0Hq/OJkFa4BiPpO2Yac4oD+ejcS5j8suzY3+VuyktUw53GTl8N6WWG6M1kMuPkz ltWcb9a4iGtD+qkbdpHhrwL6mydgrXcViXK3Ks7z/nl2871agZa/Kehojyw0VF52p3gW tX48wr0xVAjk+a/ZL39HAc9UN0z51c3ukg8qdMV1ZKH0FYQNhceBLuqgPaCblxfH4sTb JqbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=f/ZbN5quD6ggV2+4j9lRbjM6zktg3vb1d35PK4/3tT4=; b=uKTTl3AFIV37rq/EDAl/hW4iXzmBl+3+n5f+MaZSgMfvn7+iyeyOt1Oj2gV5egQBFG qfHWP2RsY64BLsBjBrGKWbm79kolSwWhQepZHUFFNGeJfDZ6IYq1+lJIinvyArDlFoi8 b6pCG4v6HToKUWMq/T+S1KTARyAZfDak3uiwUI9VM+KglAbyA2pmAbH2LaOxQTX7Ommr LTujGpUU3njxq/4BLN7fCQ6sl04E4Gs0r9XUHxcUcRNaq/Rg6LpOwqS4NZhtAlLq8kDv APquy5wjTqUM5gyYlnKL5Vh5FamoybSA8s77nkpZJAAGggs/daXXyAlEkHvXeEPbdxXG 2R0w== X-Gm-Message-State: AOAM532WmS079NFZMRS6pEmTBOqk7iUuiYyc88++eWmvvUiTs+4bZKwB R/cMa6S0q5FUvCMynE2fk9Ws/NA7np8= X-Google-Smtp-Source: ABdhPJxJTzD97ZxwPrdT84gTJ6ncO+prUg6L+3wWt+NtgvfqQ/pD0cuAUUaFXK/B8s6FnaYG/M4Zgg== X-Received: by 2002:adf:eb86:0:b0:1e6:8c92:af6b with SMTP id t6-20020adfeb86000000b001e68c92af6bmr2969735wrn.116.1648097920369; Wed, 23 Mar 2022 21:58:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l13-20020a05600002ad00b00203d64c5289sm1697902wry.112.2022.03.23.21.58.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:39 -0700 (PDT) Message-Id: <8f1b01c9ca0aff1412657d230f58fdcd7b3aeade.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:23 +0000 Subject: [PATCH v3 08/11] test-lib-functions: add parsing helpers for ls-files and ls-tree Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Several tests use awk to parse OIDs from the output of 'git ls-files --stage' and 'git ls-tree'. Introduce helpers to centralize these uses of awk. Update t5317-pack-objects-filter-objects.sh to use the new ls-files helper so that it has some usages to review. Other updates are left for the future. Signed-off-by: Neeraj Singh --- t/t5317-pack-objects-filter-objects.sh | 91 +++++++++++++------------- t/test-lib-functions.sh | 10 +++ 2 files changed, 54 insertions(+), 47 deletions(-) diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 33b740ce628..bb633c9b099 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -10,9 +10,6 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME # Test blob:none filter. test_expect_success 'setup r1' ' - echo "{print \$1}" >print_1.awk && - echo "{print \$2}" >print_2.awk && - git init r1 && for n in 1 2 3 4 5 do @@ -22,10 +19,13 @@ test_expect_success 'setup r1' ' done ' +parse_verify_pack_blob_oid () { + awk '{print $1}' - +} + test_expect_success 'verify blob count in normal packfile' ' - git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 | + test_parse_ls_files_stage_oids | sort >expected && git -C r1 pack-objects --revs --stdout >all.pack <<-EOF && @@ -35,7 +35,7 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r1 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -54,12 +54,12 @@ test_expect_success 'verify blob:none packfile has no blobs' ' test_expect_success 'verify normal and blob:none packfiles have same commits/trees' ' git -C r1 verify-pack -v ../all.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >expected && git -C r1 verify-pack -v ../filter.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -123,8 +123,8 @@ test_expect_success 'setup r2' ' ' test_expect_success 'verify blob count in normal packfile' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout >all.pack <<-EOF && @@ -134,7 +134,7 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r2 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -161,8 +161,8 @@ test_expect_success 'verify blob:limit=1000' ' ' test_expect_success 'verify blob:limit=1001' ' - git -C r2 ls-files -s large.1000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=1001 >filter.pack <<-EOF && @@ -172,15 +172,15 @@ test_expect_success 'verify blob:limit=1001' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify blob:limit=10001' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=10001 >filter.pack <<-EOF && @@ -190,15 +190,15 @@ test_expect_success 'verify blob:limit=10001' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify blob:limit=1k' ' - git -C r2 ls-files -s large.1000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=1k >filter.pack <<-EOF && @@ -208,15 +208,15 @@ test_expect_success 'verify blob:limit=1k' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify explicitly specifying oversized blob in input' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && echo HEAD >objects && @@ -226,15 +226,15 @@ test_expect_success 'verify explicitly specifying oversized blob in input' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify blob:limit=1m' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=1m >filter.pack <<-EOF && @@ -244,7 +244,7 @@ test_expect_success 'verify blob:limit=1m' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -253,12 +253,12 @@ test_expect_success 'verify blob:limit=1m' ' test_expect_success 'verify normal and blob:limit packfiles have same commits/trees' ' git -C r2 verify-pack -v ../all.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >expected && git -C r2 verify-pack -v ../filter.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -289,9 +289,8 @@ test_expect_success 'setup r3' ' ' test_expect_success 'verify blob count in normal packfile' ' - git -C r3 ls-files -s sparse1 sparse2 dir1/sparse1 dir1/sparse2 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r3 ls-files -s sparse1 sparse2 dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r3 pack-objects --revs --stdout >all.pack <<-EOF && @@ -301,7 +300,7 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r3 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -342,9 +341,8 @@ test_expect_success 'setup r4' ' ' test_expect_success 'verify blob count in normal packfile' ' - git -C r4 ls-files -s pattern sparse1 sparse2 dir1/sparse1 dir1/sparse2 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r4 ls-files -s pattern sparse1 sparse2 dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r4 pack-objects --revs --stdout >all.pack <<-EOF && @@ -354,19 +352,19 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r4 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify sparse:oid=OID' ' - git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r4 ls-files -s pattern >staged && - oid=$(awk -f print_2.awk staged) && + oid=$(test_parse_ls_files_stage_oids filter.pack <<-EOF && HEAD EOF @@ -374,15 +372,15 @@ test_expect_success 'verify sparse:oid=OID' ' git -C r4 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify sparse:oid=oid-ish' ' - git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r4 pack-objects --revs --stdout --filter=sparse:oid=main:pattern >filter.pack <<-EOF && @@ -392,7 +390,7 @@ test_expect_success 'verify sparse:oid=oid-ish' ' git -C r4 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -402,9 +400,8 @@ test_expect_success 'verify sparse:oid=oid-ish' ' # This models previously omitted objects that we did not receive. test_expect_success 'setup r1 - delete loose blobs' ' - git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 | + test_parse_ls_files_stage_oids | sort >expected && for id in `cat expected | sed "s|..|&/|"` diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index a027f0c409e..e6011409e2f 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1782,6 +1782,16 @@ test_oid_to_path () { echo "${1%$basename}/$basename" } +# Parse oids from git ls-files --staged output +test_parse_ls_files_stage_oids () { + awk '{print $2}' - +} + +# Parse oids from git ls-tree output +test_parse_ls_tree_oids () { + awk '{print $3}' - +} + # Choose a port number based on the test script's number and store it in # the given variable name, unless that variable already contains a number. test_set_port () { From patchwork Thu Mar 24 04:58:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3847DC433F5 for ; Thu, 24 Mar 2022 04:59:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345265AbiCXFAv (ORCPT ); Thu, 24 Mar 2022 01:00:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345810AbiCXFA3 (ORCPT ); Thu, 24 Mar 2022 01:00:29 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D1E82C669 for ; Wed, 23 Mar 2022 21:58:43 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id b19so4913833wrh.11 for ; Wed, 23 Mar 2022 21:58:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fVqX14TjGWGRDpi6324fkA+5n46rvcugh6p5qU84Jk8=; b=c1GRz33kgKHa1NefEII5us34Nzpe3VR7AWNdOpWXed56B2gpHUIwR450FxUeQnHD7x 3ArYnkqJ6KdesAdgnaKYNVTEozczipk+6r2uZNxH+8OiNvQLLbkKAV0KrWLgcymV59AQ g9idETfaN8LkkSTpBUMKt9M8Sq5ntfVD4vLj8d2CQPqbTKUtFZa02QHaznRLpsNfCVnk qpXN1aGYZBaimWy9/5+yujZ4O1N9C6+RI2tmbwg7IFOCInGdjKaE1wVwCwvgv67DpixG c74L8kHTr7C5ytwQoHzaYNv2OBmj0mgBL6VXkFdDtJ4jeEjKXMiFRVU0sY/660UNh5vu qbxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fVqX14TjGWGRDpi6324fkA+5n46rvcugh6p5qU84Jk8=; b=Iaw8K/9QY6FBZ9CMBNkv3GVRxiLBuXRuLMIpvQ1hM/WnK+jmP+oRxUIs2+6T+9ez1Z RlOvzqUBO6px504DRgx2cpkvxj5fqPzNyTdJ65GyLHgICZGPKjxboXNbMkWAG0SCHUHS 0WkgThchuU7+DyS/ZdNaRM1E1vZGTg4LU3fGby9c4qPMpG7IQvG+ZbcQJKr2ng5M78m4 xTmkWtE86eS0D+stOO09nugizLmYInMsbyhZVfeGkxQbe+NtLDIK1WXQhnAGENfjLBFH ugsJlVBCkVhLwwsiW+GjhVpDr62ku45/T4WJPWyZzKqM2/LTbpgO56kGdPdI80BLCSpT srhQ== X-Gm-Message-State: AOAM532JBb7IoXLv24+PiV4EAItFUbJ+2lnebSeIC4rHRj2imxxCwALH clAJcYjNDBWaAdSMgKX3eBmZddwrpmo= X-Google-Smtp-Source: ABdhPJy+aEqQ4NRSB4f/5nWz0Z1L1FeRv+mArvjaSVtV4aGqhg6vQZt9llLjXaD9fy9wg0hCJhzVyA== X-Received: by 2002:a5d:48d1:0:b0:1e3:2401:f229 with SMTP id p17-20020a5d48d1000000b001e32401f229mr2851546wrs.694.1648097921990; Wed, 23 Mar 2022 21:58:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c5-20020a5d63c5000000b002040822b680sm2301091wrw.81.2022.03.23.21.58.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:41 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:24 +0000 Subject: [PATCH v3 09/11] core.fsyncmethod: tests for batch mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Add test cases to exercise batch mode for: * 'git add' * 'git stash' * 'git update-index' * 'git unpack-objects' These tests ensure that the added data winds up in the object database. In this change we introduce a new test helper lib-unique-files.sh. The goal of this library is to create a tree of files that have different oids from any other files that may have been created in the current test repo. This helps us avoid missing validation of an object being added due to it already being in the repo. Signed-off-by: Neeraj Singh --- t/lib-unique-files.sh | 32 ++++++++++++++++++++++++++++++++ t/t3700-add.sh | 28 ++++++++++++++++++++++++++++ t/t3903-stash.sh | 20 ++++++++++++++++++++ t/t5300-pack-object.sh | 41 +++++++++++++++++++++++++++-------------- 4 files changed, 107 insertions(+), 14 deletions(-) create mode 100644 t/lib-unique-files.sh diff --git a/t/lib-unique-files.sh b/t/lib-unique-files.sh new file mode 100644 index 00000000000..74efca91dd7 --- /dev/null +++ b/t/lib-unique-files.sh @@ -0,0 +1,32 @@ +# Helper to create files with unique contents + +# Create multiple files with unique contents within this test run. Takes the +# number of directories, the number of files in each directory, and the base +# directory. +# +# test_create_unique_files 2 3 my_dir -- Creates 2 directories with 3 files +# each in my_dir, all with contents +# different from previous invocations +# of this command in this run. + +test_create_unique_files () { + test "$#" -ne 3 && BUG "3 param" + + local dirs="$1" && + local files="$2" && + local basedir="$3" && + local counter=0 && + test_tick && + local basedata=$basedir$test_tick && + rm -rf "$basedir" && + for i in $(test_seq $dirs) + do + local dir=$basedir/dir$i && + mkdir -p "$dir" && + for j in $(test_seq $files) + do + counter=$((counter + 1)) && + echo "$basedata.$counter">"$dir/file$j.txt" + done + done +} diff --git a/t/t3700-add.sh b/t/t3700-add.sh index b1f90ba3250..8979c8a5f03 100755 --- a/t/t3700-add.sh +++ b/t/t3700-add.sh @@ -8,6 +8,8 @@ test_description='Test of git add, including the -- option.' TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh +. $TEST_DIRECTORY/lib-unique-files.sh + # Test the file mode "$1" of the file "$2" in the index. test_mode_in_index () { case "$(git ls-files -s "$2")" in @@ -34,6 +36,32 @@ test_expect_success \ 'Test that "git add -- -q" works' \ 'touch -- -q && git add -- -q' +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'git add: core.fsyncmethod=batch' " + test_create_unique_files 2 4 files_base_dir1 && + GIT_TEST_FSYNC=1 git $BATCH_CONFIGURATION add -- ./files_base_dir1/ && + git ls-files --stage files_base_dir1/ | + test_parse_ls_files_stage_oids >added_files_oids && + + # We created 2 subdirs with 4 files each (8 files total) above + test_line_count = 8 added_files_oids && + git cat-file --batch-check='%(objectname)' added_files_actual && + test_cmp added_files_oids added_files_actual +" + +test_expect_success 'git update-index: core.fsyncmethod=batch' " + test_create_unique_files 2 4 files_base_dir2 && + find files_base_dir2 ! -type d -print | xargs git $BATCH_CONFIGURATION update-index --add -- && + git ls-files --stage files_base_dir2 | + test_parse_ls_files_stage_oids >added_files2_oids && + + # We created 2 subdirs with 4 files each (8 files total) above + test_line_count = 8 added_files2_oids && + git cat-file --batch-check='%(objectname)' added_files2_actual && + test_cmp added_files2_oids added_files2_actual +" + test_expect_success \ 'git add: Test that executable bit is not used if core.filemode=0' \ 'git config core.filemode 0 && diff --git a/t/t3903-stash.sh b/t/t3903-stash.sh index 4abbc8fccae..20e94881964 100755 --- a/t/t3903-stash.sh +++ b/t/t3903-stash.sh @@ -9,6 +9,7 @@ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./test-lib.sh +. $TEST_DIRECTORY/lib-unique-files.sh test_expect_success 'usage on cmd and subcommand invalid option' ' test_expect_code 129 git stash --invalid-option 2>usage && @@ -1410,6 +1411,25 @@ test_expect_success 'stash handles skip-worktree entries nicely' ' git rev-parse --verify refs/stash:A.t ' + +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'stash with core.fsyncmethod=batch' " + test_create_unique_files 2 4 files_base_dir && + GIT_TEST_FSYNC=1 git $BATCH_CONFIGURATION stash push -u -- ./files_base_dir/ && + + # The files were untracked, so use the third parent, + # which contains the untracked files + git ls-tree -r stash^3 -- ./files_base_dir/ | + test_parse_ls_tree_oids >stashed_files_oids && + + # We created 2 dirs with 4 files each (8 files total) above + test_line_count = 8 stashed_files_oids && + git cat-file --batch-check='%(objectname)' stashed_files_actual && + test_cmp stashed_files_oids stashed_files_actual +" + + test_expect_success 'git stash succeeds despite directory/file change' ' test_create_repo directory_file_switch_v1 && ( diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index a11d61206ad..f8a0f309e2d 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -161,22 +161,27 @@ test_expect_success 'pack-objects with bogus arguments' ' ' check_unpack () { + local packname="$1" && + local object_list="$2" && + local git_config="$3" && test_when_finished "rm -rf git2" && - git init --bare git2 && - git -C git2 unpack-objects -n <"$1".pack && - git -C git2 unpack-objects <"$1".pack && - (cd .git && find objects -type f -print) | - while read path - do - cmp git2/$path .git/$path || { - echo $path differs. - return 1 - } - done + git $git_config init --bare git2 && + ( + git $git_config -C git2 unpack-objects -n <"$packname".pack && + git $git_config -C git2 unpack-objects <"$packname".pack && + git $git_config -C git2 cat-file --batch-check="%(objectname)" + ) <"$object_list" >current && + cmp "$object_list" current } test_expect_success 'unpack without delta' ' - check_unpack test-1-${packname_1} + check_unpack test-1-${packname_1} obj-list +' + +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'unpack without delta (core.fsyncmethod=batch)' ' + check_unpack test-1-${packname_1} obj-list "$BATCH_CONFIGURATION" ' test_expect_success 'pack with REF_DELTA' ' @@ -185,7 +190,11 @@ test_expect_success 'pack with REF_DELTA' ' ' test_expect_success 'unpack with REF_DELTA' ' - check_unpack test-2-${packname_2} + check_unpack test-2-${packname_2} obj-list +' + +test_expect_success 'unpack with REF_DELTA (core.fsyncmethod=batch)' ' + check_unpack test-2-${packname_2} obj-list "$BATCH_CONFIGURATION" ' test_expect_success 'pack with OFS_DELTA' ' @@ -195,7 +204,11 @@ test_expect_success 'pack with OFS_DELTA' ' ' test_expect_success 'unpack with OFS_DELTA' ' - check_unpack test-3-${packname_3} + check_unpack test-3-${packname_3} obj-list +' + +test_expect_success 'unpack with OFS_DELTA (core.fsyncmethod=batch)' ' + check_unpack test-3-${packname_3} obj-list "$BATCH_CONFIGURATION" ' test_expect_success 'compare delta flavors' ' From patchwork Thu Mar 24 04:58:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C2BEC433F5 for ; Thu, 24 Mar 2022 04:59:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346232AbiCXFAs (ORCPT ); Thu, 24 Mar 2022 01:00:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346052AbiCXFAa (ORCPT ); Thu, 24 Mar 2022 01:00:30 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 66B542DD4C for ; Wed, 23 Mar 2022 21:58:44 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id j18so4936805wrd.6 for ; Wed, 23 Mar 2022 21:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NZ9DwWe8RMi6qei+a97iaf/fWz4PMyQi0ZAFeh+epW8=; b=W8xn27m74ZgiygeP4RLKH1B5NnOrlDq0KEDLj9vZlGLH9FD12QAd7y9IkbRehDRHI4 VLg0gFnogo5uSvW7l2qqcKJ1ihQJGPdLI11CUZYVrkFI8QskDR+gAF5GO7BX+k1+wQcY s7pD7JfyTpNjNrFVUtN6jTTw7zVoTzfYPmaHYqhUDMAO6QowgqZtSVheBhTUywDeRPqT 504R5NyUqDjG+ZiWIbfgI1QpxJ1zELoZYosXpW31Jjk95nqEXStiJv6uaRkVdK3MfXlq QdRldRLD7kod5AdXBA13jok4Nh8xBYrryN1AR+1YefwxdJ5Ud/Ie+8sVedRMFViF/PiA /17A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NZ9DwWe8RMi6qei+a97iaf/fWz4PMyQi0ZAFeh+epW8=; b=x3VRgb/Cwpej+0Kl+0Bw5PEY07G3JEf/DJRwqEjEKa/nvLCWIhj8qk8ADKhaS8hHPX kGicvW2shUVZaXFS7ERuYgMP/oUpwLZfxZlSmdTi89mRSetdL2JEyB06zXHlO9/JDwzj ugsvwWxG86GDCnJRtolyEFicoU1HYoStRA3eGMzYQqHlycuzBYnvwu7qJDX2qFwcFJL4 DDJeCHThdjSEFm+fX8Ed3jDQt8ig0lPMElYniXWL52RhaAQJHQNXZpR/APPBOFiA/AQQ jNppXrTkMJiiWenuyjBtGjQoIpACl7aKI3bczPdB78rVvmBKhnwzoedd4zxUZf5VxX1F yk3g== X-Gm-Message-State: AOAM532vCVuZbq2hTfAgGlXyU5LTjWYDt9Dri99p4/TgYQRswkzOhXuC tvb3UTojKSCuepwD7FQrq9Ub4izrwCM= X-Google-Smtp-Source: ABdhPJx25c/mX4sw29e+3Vac9qU5ofV70qS+/OvAWDN5/NphhdGi/mHcFwdwObIIuk+KCRxhnSN0Gg== X-Received: by 2002:a05:6000:3cb:b0:205:7b9d:cade with SMTP id b11-20020a05600003cb00b002057b9dcademr2905925wrg.239.1648097923190; Wed, 23 Mar 2022 21:58:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n22-20020a05600c4f9600b0038c6ec42c38sm1137854wmq.6.2022.03.23.21.58.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:42 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:25 +0000 Subject: [PATCH v3 10/11] core.fsyncmethod: performance tests for add and stash Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Add basic performance tests for "git add" and "git stash" of a lot of new objects with various fsync settings. This shows the benefit of batch mode relative to full fsync. Signed-off-by: Neeraj Singh --- t/perf/p3700-add.sh | 59 ++++++++++++++++++++++++++++++++++++++++ t/perf/p3900-stash.sh | 62 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+) create mode 100755 t/perf/p3700-add.sh create mode 100755 t/perf/p3900-stash.sh diff --git a/t/perf/p3700-add.sh b/t/perf/p3700-add.sh new file mode 100755 index 00000000000..2ea78c9449d --- /dev/null +++ b/t/perf/p3700-add.sh @@ -0,0 +1,59 @@ +#!/bin/sh +# +# This test measures the performance of adding new files to the object database +# and index. The test was originally added to measure the effect of the +# core.fsyncMethod=batch mode, which is why we are testing different values +# of that setting explicitly and creating a lot of unique objects. + +test_description="Tests performance of add" + +# Fsync is normally turned off for the test suite. +GIT_TEST_FSYNC=1 +export GIT_TEST_FSYNC + +. ./perf-lib.sh + +. $TEST_DIRECTORY/lib-unique-files.sh + +test_perf_default_repo +test_checkout_worktree + +dir_count=10 +files_per_dir=50 +total_files=$((dir_count * files_per_dir)) + +# We need to create the files each time we run the perf test, but +# we do not want to measure the cost of creating the files, so run +# the test once. +if test "${GIT_PERF_REPEAT_COUNT-1}" -ne 1 +then + echo "warning: Setting GIT_PERF_REPEAT_COUNT=1" >&2 + GIT_PERF_REPEAT_COUNT=1 +fi + +for m in false true batch +do + test_expect_success "create the files for object_fsyncing=$m" ' + git reset --hard && + # create files across directories + test_create_unique_files $dir_count $files_per_dir files + ' + + case $m in + false) + FSYNC_CONFIG='-c core.fsync=-loose-object -c core.fsyncmethod=fsync' + ;; + true) + FSYNC_CONFIG='-c core.fsync=loose-object -c core.fsyncmethod=fsync' + ;; + batch) + FSYNC_CONFIG='-c core.fsync=loose-object -c core.fsyncmethod=batch' + ;; + esac + + test_perf "add $total_files files (object_fsyncing=$m)" " + git $FSYNC_CONFIG add files + " +done + +test_done diff --git a/t/perf/p3900-stash.sh b/t/perf/p3900-stash.sh new file mode 100755 index 00000000000..3526f06cef4 --- /dev/null +++ b/t/perf/p3900-stash.sh @@ -0,0 +1,62 @@ +#!/bin/sh +# +# This test measures the performance of adding new files to the object database +# and index. The test was originally added to measure the effect of the +# core.fsyncMethod=batch mode, which is why we are testing different values +# of that setting explicitly and creating a lot of unique objects. + +test_description="Tests performance of stash" + +# Fsync is normally turned off for the test suite. +GIT_TEST_FSYNC=1 +export GIT_TEST_FSYNC + +. ./perf-lib.sh + +. $TEST_DIRECTORY/lib-unique-files.sh + +test_perf_default_repo +test_checkout_worktree + +dir_count=10 +files_per_dir=50 +total_files=$((dir_count * files_per_dir)) + +# We need to create the files each time we run the perf test, but +# we do not want to measure the cost of creating the files, so run +# the test once. +if test "${GIT_PERF_REPEAT_COUNT-1}" -ne 1 +then + echo "warning: Setting GIT_PERF_REPEAT_COUNT=1" >&2 + GIT_PERF_REPEAT_COUNT=1 +fi + +for m in false true batch +do + test_expect_success "create the files for object_fsyncing=$m" ' + git reset --hard && + # create files across directories + test_create_unique_files $dir_count $files_per_dir files + ' + + case $m in + false) + FSYNC_CONFIG='-c core.fsync=-loose-object -c core.fsyncmethod=fsync' + ;; + true) + FSYNC_CONFIG='-c core.fsync=loose-object -c core.fsyncmethod=fsync' + ;; + batch) + FSYNC_CONFIG='-c core.fsync=loose-object -c core.fsyncmethod=batch' + ;; + esac + + # We only stash files in the 'files' subdirectory since + # the perf test infrastructure creates files in the + # current working directory that need to be preserved + test_perf "stash $total_files files (object_fsyncing=$m)" " + git $FSYNC_CONFIG stash push -u -- files + " +done + +test_done From patchwork Thu Mar 24 04:58:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12790366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B32E7C433EF for ; Thu, 24 Mar 2022 04:59:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346844AbiCXFAv (ORCPT ); Thu, 24 Mar 2022 01:00:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345226AbiCXFAa (ORCPT ); Thu, 24 Mar 2022 01:00:30 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 723882DD6F for ; Wed, 23 Mar 2022 21:58:46 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id t11so4943222wrm.5 for ; Wed, 23 Mar 2022 21:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/dJLnMNPf593DDA+ox52qw4qYfm4mKhrA2SM6fu1eys=; b=KP4cHSFMBx+l9IVhPuUwBeNIKsyUNAfxdT5j2ZM4IR3Z1meDkaTYeMWbyKhJga9KNo 5akPQi1q1ZWKrF12Ja43VPfQAOUSYhHS9dW1yba8uUdITr9yid+vasgUkf0+sYtr2IRe 1z/fxtLUrB6FKNU3IN1RKSBlp02GKb6aTRn3tahXTk+rcLNJMl0V/N+BRPHYGTty9G0j sTdLTzTln3XMsJwbwPvbt5O2ijSe5iuwy+O9Xypnlr+NwY+w56mvXkuezZAF6dj50mAt lnMkPytHc18jx4TOTyVOCa0pmy1rZuNg41oos01lhvJlGcqqEY6NygEiofgVyp8ypIcU PWnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/dJLnMNPf593DDA+ox52qw4qYfm4mKhrA2SM6fu1eys=; b=4L3AxOeFUOMf9lM8sclL5Vwj8wlIKxgLycV3c8qpjiJCZGRQQ0JmNfbmsN8UINmE/j RegKO7h5E6wWlSS4FIc1T2vpT6mLan8ixfcuttoJ/W7yXUFxn5KXgOqYK93f3v7eRJjU tvd+8RjN2F7OLaGLmaQ88feVvBMTQaG1XiyJijkDwx/VAs2xcFq/TwzebvVnMh6GrjZX Gf3G89V0dQQHG8jf79bjPooaX9SYyiPMGCmjEuRy5A/VD8hCs8gW9tXlNiiwEpJK17fj WeADIe5fsY73wFr4vhrvBQ7oL9HN7Uw9H+Vdqc9WgvENu3hP28HW/FxgWakOup3wEjbr zVXA== X-Gm-Message-State: AOAM5308Bs/hEQNoMUXuRj7SqukuDWnpFnLw9/VW8m1M8p5REsZ7jwT3 VNuOT6q/crLG6rFTyc4a/XEnkzYIGjg= X-Google-Smtp-Source: ABdhPJxSVGPbdNW6nR6vanvQu8ggIBRIKI7rXxMDrWrfjr0OdQgqa7fjn34W5EFCy75gBt/DGky1UQ== X-Received: by 2002:adf:a302:0:b0:1ed:bf30:40e3 with SMTP id c2-20020adfa302000000b001edbf3040e3mr2854332wrb.270.1648097924626; Wed, 23 Mar 2022 21:58:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w7-20020a1cf607000000b00389a5390180sm1109269wmc.25.2022.03.23.21.58.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 21:58:43 -0700 (PDT) Message-Id: <6b832e89bc47f00af4bd186b71577409914970f7.1648097906.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 24 Mar 2022 04:58:26 +0000 Subject: [PATCH v3 11/11] core.fsyncmethod: correctly camel-case warning message Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, Bagas Sanjaya , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The warning for an unrecognized fsyncMethod was not camel-cased. Signed-off-by: Neeraj Singh --- config.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config.c b/config.c index e9cac5f4707..ae819dee20b 100644 --- a/config.c +++ b/config.c @@ -1697,7 +1697,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) if (!strcmp(var, "core.fsyncobjectfiles")) { if (fsync_object_files < 0) - warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); + warning(_("core.fsyncObjectFiles is deprecated; use core.fsync instead")); fsync_object_files = git_config_bool(var, value); return 0; }