From patchwork Wed Mar 30 05:05:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2205C433F5 for ; Wed, 30 Mar 2022 05:05:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242723AbiC3FH3 (ORCPT ); Wed, 30 Mar 2022 01:07:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235117AbiC3FHX (ORCPT ); Wed, 30 Mar 2022 01:07:23 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE21265D32 for ; Tue, 29 Mar 2022 22:05:38 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id p189so11514776wmp.3 for ; Tue, 29 Mar 2022 22:05:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=VcBruSMhra7Hu3xR7auyp+juizXtAATaYi6To93RDbY=; b=D6PK0v7wWm7l2Oy6scqAXta0jJDYPpR7rvBXueypZCYXitZNTf+/XbS1OVRal0wu9Q dDPC/vym99c97wQlHGfSOXAu76RibZH8wGVO7fD7TJrnFXoNv3foHCBBxvz4v6kHy79W ZMkf9Kg2QCuGeCHe8RmjaRICvsczZTyXlWzJoVuMgTF1rFt2hQEdjlHrZ4+hWL6Qyi58 ct4JURoCdArTtAWjLEUaNlVCAi/F+JYeUeuEznioVMJGa/ReGJ5v4fhRGSc5YyCYHHpV A+ZnsOn1kps9FjDDMow4dRKqYqyykdZ0pAwoK9zfYXvL3moGGRu/sMds3MpgI/RDKUGY 4wtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=VcBruSMhra7Hu3xR7auyp+juizXtAATaYi6To93RDbY=; b=W4gMkh8QotOb4eV5Uq8EdJ9ju9G5oU5VIs4FPAV/ZiZWAMNWmMwJVq+l2vzjg/WMjQ ceVQPVtYHulyMr8hkHM0XCnWPi6c6BwGkXqcFGVeKdIgEnaTFKVhGTzuB4/ntot/fSdn eqLyztsIndvdzbQctaMOOI3BNS7jjlNEaQ1jnagUNdHcUfxjP8RaiFqy4wVL44+za5o9 ihDkNEMhj5wB0GFLtRPbpsCxfb7fWPQVMbI5t0xlSUr4/r1ngIHsgyVlZwpdfbkJdhAn K27HxCUOMDG2ofUTpdzc1eiFX+dxX4ik9ZHuDpDk0DPnxF4PMR4cHwjQBk09Dpk3A3Tk hb6A== X-Gm-Message-State: AOAM533sBDRUJetEfxjgdWMZIk+/VHafW8DHpBvMjG2ARTYaY6nufwpA NpFHOTdvMBMckfjbfG6efhYUlbqUbm4= X-Google-Smtp-Source: ABdhPJy/r4tVODtKs4tTaLiR9hooasSjV6Dlt6d8Oe4BJs4TuzMpKAQc9bpt+pWg4zwusx/yy116ng== X-Received: by 2002:a1c:f415:0:b0:37f:ab4d:1df2 with SMTP id z21-20020a1cf415000000b0037fab4d1df2mr2553562wma.75.1648616736830; Tue, 29 Mar 2022 22:05:36 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b0038cb84f4218sm3869791wmq.27.2022.03.29.22.05.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:36 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:19 +0000 Subject: [PATCH v5 01/14] bulk-checkin: rename 'state' variable and separate 'plugged' boolean Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit prepares for adding batch-fsync to the bulk-checkin infrastructure. The bulk-checkin infrastructure is currently used to batch up addition of large blobs to a packfile. When a blob is larger than big_file_threshold, we unconditionally add it to a pack. If bulk checkins are 'plugged', we allow multiple large blobs to be added to a single pack until we reach the packfile size limit; otherwise, we simply make a new packfile for each large blob. The 'unplug' call tells us when the series of blob additions is done so that we can finish the packfiles and make their objects available to subsequent operations. Stated another way, bulk-checkin allows callers to define a transaction that adds multiple objects to the object database, where the object database can optimize its internal operations within the transaction boundary. Batched fsync will fit into bulk-checkin by taking advantage of the plug/unplug functionality to determine the appropriate time to fsync and make newly-added objects available in the primary object database. * Rename 'state' variable to 'bulk_checkin_state', since we will later be adding 'bulk_fsync_objdir'. This also makes the variable easier to find in the debugger, since the name is more unique. * Move the 'plugged' data member of 'bulk_checkin_state' into a separate static variable. Doing this avoids resetting the variable in finish_bulk_checkin when zeroing the 'bulk_checkin_state'. As-is, we seem to unintentionally disable the plugging functionality the first time a new packfile must be created due to packfile size limits. While disabling the plugging state only results in suboptimal behavior for the current code, it would be fatal for the bulk-fsync functionality later in this patch series. Signed-off-by: Neeraj Singh --- bulk-checkin.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/bulk-checkin.c b/bulk-checkin.c index 6d6c37171c9..577b135e39c 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -10,9 +10,9 @@ #include "packfile.h" #include "object-store.h" -static struct bulk_checkin_state { - unsigned plugged:1; +static int bulk_checkin_plugged; +static struct bulk_checkin_state { char *pack_tmp_name; struct hashfile *f; off_t offset; @@ -21,7 +21,7 @@ static struct bulk_checkin_state { struct pack_idx_entry **written; uint32_t alloc_written; uint32_t nr_written; -} state; +} bulk_checkin_state; static void finish_tmp_packfile(struct strbuf *basename, const char *pack_tmp_name, @@ -278,21 +278,23 @@ int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags) { - int status = deflate_to_pack(&state, oid, fd, size, type, + int status = deflate_to_pack(&bulk_checkin_state, oid, fd, size, type, path, flags); - if (!state.plugged) - finish_bulk_checkin(&state); + if (!bulk_checkin_plugged) + finish_bulk_checkin(&bulk_checkin_state); return status; } void plug_bulk_checkin(void) { - state.plugged = 1; + assert(!bulk_checkin_plugged); + bulk_checkin_plugged = 1; } void unplug_bulk_checkin(void) { - state.plugged = 0; - if (state.f) - finish_bulk_checkin(&state); + assert(bulk_checkin_plugged); + bulk_checkin_plugged = 0; + if (bulk_checkin_state.f) + finish_bulk_checkin(&bulk_checkin_state); } From patchwork Wed Mar 30 05:05:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87B52C433EF for ; Wed, 30 Mar 2022 05:05:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242715AbiC3FHZ (ORCPT ); Wed, 30 Mar 2022 01:07:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242708AbiC3FHX (ORCPT ); Wed, 30 Mar 2022 01:07:23 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A938266213 for ; Tue, 29 Mar 2022 22:05:39 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id d7so27561665wrb.7 for ; Tue, 29 Mar 2022 22:05:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=TOI+vuPMSYdUksjZQAWmkRtVe6MxK9DxmcwWwfN9fG4=; b=PM2ZfgfmgkRYnWj9rXgXbluxCMIRlegPv604c/Icn9VHL44pgSPp9Ks+9Q/hBdv3TU yXSaTeyRj+pThKOC+y9R1SeJxWYt2j+13KQMYP7mlvdm3q35hgRsp1hxsJCpQ5xtb5rv Hj/+Npmsg51+DRAqDaoszRi2uVKqjMZLce/WdlvYEqt6esVuCe0ECpe0bQirXO9Xrhg+ EuWmJeNdemGfMIwg9UR/Y1DqWAB5CyJHzXMhnxP2eN3H8663QfUPbR3lAM0Noz4MvxD8 aKaxCwuP/kPCq6yFsOoqDA59We5T0bkoSw6o9s5FCZbyo5a2BrQeG5QLehMZFeHcvhOG G2vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=TOI+vuPMSYdUksjZQAWmkRtVe6MxK9DxmcwWwfN9fG4=; b=ahi56Nil8GaWM4W5PYiPxst/v+0e31L2VupEvl5G6h98DFBh4VBSvKOJr6C/J/L1pw 1HPL0kZisNl+f6poQBra3DiQ7hkqT5CP2bbQulkPTebpLC1PUX6W5mx65JS3uoZx9Pfz nUDrzaOpwvt6JYvP9NKVyl6bijZDWOexHDc+h/ks7I9ZbcpUfNmTuZvzRYnydHNgIRdn B6b2/AO2jxK45cO9pGuB8AUbGxI3LNUxJaVFs6iaqxAAFsiH+wbCnWEXPsdz+IREdGtt 3Y21+J2nYkrpPZygtgTe2+JG3R8+ktR1z/ELDYOIPgINIAtcMtQYEAzUyFKMt1WQ7tUK Vv8A== X-Gm-Message-State: AOAM531IH/+v/+UedIaV2O+CvpO2IIClGrzBsrpPig1V/ifQzF4M24Pe kzfkD3+LrB5ywKeEEU7G54Z3VoCBwIk= X-Google-Smtp-Source: ABdhPJzPuw7+QtulELCsaEiT07sI59usQ85YEMOaoHyEWZIB4nMhq0NZmz7OvVpgCseNzq52ULm5zA== X-Received: by 2002:a5d:4a0f:0:b0:1ed:d377:288a with SMTP id m15-20020a5d4a0f000000b001edd377288amr33213129wrq.3.1648616737942; Tue, 29 Mar 2022 22:05:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p23-20020a1c5457000000b0038c98c12ea9sm3721386wmi.1.2022.03.29.22.05.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:37 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:20 +0000 Subject: [PATCH v5 02/14] bulk-checkin: rebrand plug/unplug APIs as 'odb transactions' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Make it clearer in the naming and documentation of the plug_bulk_checkin and unplug_bulk_checkin APIs that they can be thought of as a "transaction" to optimize operations on the object database. These transactions may be nested so that subsystems like the cache-tree writing code can optimize their operations without caring whether the top-level code has a transaction active. Signed-off-by: Neeraj Singh --- builtin/add.c | 4 ++-- bulk-checkin.c | 20 ++++++++++++-------- bulk-checkin.h | 14 ++++++++++++-- 3 files changed, 26 insertions(+), 12 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index 3ffb86a4338..9bf37ceae8e 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -670,7 +670,7 @@ int cmd_add(int argc, const char **argv, const char *prefix) string_list_clear(&only_match_skip_worktree, 0); } - plug_bulk_checkin(); + begin_odb_transaction(); if (add_renormalize) exit_status |= renormalize_tracked_files(&pathspec, flags); @@ -682,7 +682,7 @@ int cmd_add(int argc, const char **argv, const char *prefix) if (chmod_arg && pathspec.nr) exit_status |= chmod_pathspec(&pathspec, chmod_arg[0], show_only); - unplug_bulk_checkin(); + end_odb_transaction(); finish: if (write_locked_index(&the_index, &lock_file, diff --git a/bulk-checkin.c b/bulk-checkin.c index 577b135e39c..8b0fd5c7723 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -10,7 +10,7 @@ #include "packfile.h" #include "object-store.h" -static int bulk_checkin_plugged; +static int odb_transaction_nesting; static struct bulk_checkin_state { char *pack_tmp_name; @@ -280,21 +280,25 @@ int index_bulk_checkin(struct object_id *oid, { int status = deflate_to_pack(&bulk_checkin_state, oid, fd, size, type, path, flags); - if (!bulk_checkin_plugged) + if (!odb_transaction_nesting) finish_bulk_checkin(&bulk_checkin_state); return status; } -void plug_bulk_checkin(void) +void begin_odb_transaction(void) { - assert(!bulk_checkin_plugged); - bulk_checkin_plugged = 1; + odb_transaction_nesting += 1; } -void unplug_bulk_checkin(void) +void end_odb_transaction(void) { - assert(bulk_checkin_plugged); - bulk_checkin_plugged = 0; + odb_transaction_nesting -= 1; + if (odb_transaction_nesting < 0) + BUG("Unbalanced ODB transaction nesting"); + + if (odb_transaction_nesting) + return; + if (bulk_checkin_state.f) finish_bulk_checkin(&bulk_checkin_state); } diff --git a/bulk-checkin.h b/bulk-checkin.h index b26f3dc3b74..69a94422ac7 100644 --- a/bulk-checkin.h +++ b/bulk-checkin.h @@ -10,7 +10,17 @@ int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags); -void plug_bulk_checkin(void); -void unplug_bulk_checkin(void); +/* + * Tell the object database to optimize for adding + * multiple objects. end_odb_transaction must be called + * to make new objects visible. + */ +void begin_odb_transaction(void); + +/* + * Tell the object database to make any objects from the + * current transaction visible. + */ +void end_odb_transaction(void); #endif From patchwork Wed Mar 30 05:05:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3648FC433EF for ; Wed, 30 Mar 2022 05:05:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242725AbiC3FHc (ORCPT ); Wed, 30 Mar 2022 01:07:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234045AbiC3FHY (ORCPT ); Wed, 30 Mar 2022 01:07:24 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87CB06621B for ; Tue, 29 Mar 2022 22:05:40 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id c7so982350wrd.0 for ; Tue, 29 Mar 2022 22:05:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=W9Vimte0sGyR2DMe0rSIsRe6vitxNiJv2vsesuc0ht0=; b=Oh77vFaTu3RR7W4K3ofhuIo+hBGtysl+oowvms0G2uxrqZCs2uK5lg2+dA4yGjSak9 heSX0hKqrMNJ0hoUlzk2EUEZt4KRwmaTT66Pysf6ImRP7WGmCd6x9QjafFJTv0pBZV3o UpxVZ7P6K2eBkvaK1jhp0Mj6SRoUOgpjINAshD9GBEsOpLc5NZpEfrnn8cTxV9XXmWBb 7+Ma/UKBLNcNFq5lVA3lSVmcDlWUxzoZfhbHDWRss0ltQ/8iS6DSSFGetIhSserIlwQZ pIuF1OR/IJnpv/ru5192PgJBxr3p9cKPlmwt7eGOchVcZFZv+Azs3XZh9PDMisDVrPQu lqPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=W9Vimte0sGyR2DMe0rSIsRe6vitxNiJv2vsesuc0ht0=; b=CFLz8lkxhshkvs7Buoa5upV9CvkXOQqFIA6LEntUvtbfJWzMAnmm8oDGdE10jcqbFj Q1POxE9GSX8+oMbVWKNUuGdz14NI1d7IcylDps2IC2NNEDZT3zEeexNCoI7VBHIizB9N jLbpj63k/j+PnFmWf4GvfVFtlwTAYSmhx6zh19GbdzHykxUSZM3n4LfvQY0bXFtLQFrT pzSpN3/uTZnszhWYipnwmgE/DRQsgUIkjNO2iTlaQtsP1lf0D3kknpzTyxOxZ3lv9/6x MIHe86hpuqzwKp3BbbUqm7Odua1vxEGTD6dkD0Y07M9iU9RyuAUW4eBxfR4pgKyOAzUn EbyQ== X-Gm-Message-State: AOAM533AgYza+GrNfhz9AZufazIbnfSgx5wr+0/nC9ZOgPKSbY5YTqiD qQcegIbDVE6Cm6iQGcHV5+4i37/aehk= X-Google-Smtp-Source: ABdhPJyVv2riFIYlWqBUPYT86Fy6LvU/yjR2V2wQ2NHj7aPYbcJQ4D4nNe+eCQ1WANpG2UGYS7d7KQ== X-Received: by 2002:a5d:584f:0:b0:204:171b:a4bc with SMTP id i15-20020a5d584f000000b00204171ba4bcmr35887798wrf.592.1648616738855; Tue, 29 Mar 2022 22:05:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w8-20020a1cf608000000b0038c8fdc93d6sm3610464wmc.28.2022.03.29.22.05.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:38 -0700 (PDT) Message-Id: <2d1bc4568ac744f11c886a5f964dbe563c04ce8b.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:21 +0000 Subject: [PATCH v5 03/14] object-file: pass filename to fsync_or_die Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh If we die while trying to fsync a loose object file, pass the actual filename we're trying to sync. This is likely to be more helpful for a user trying to diagnose the cause of the failure than the former 'loose object file' string. It also sidesteps any concerns about translating the die message differently for loose objects versus something else that has a real path. Signed-off-by: Neeraj Singh --- object-file.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/object-file.c b/object-file.c index b254bc50d70..5ffbf3d4fd4 100644 --- a/object-file.c +++ b/object-file.c @@ -1888,16 +1888,16 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf, } /* Finalize a file on disk, and close it. */ -static void close_loose_object(int fd) +static void close_loose_object(int fd, const char *filename) { if (the_repository->objects->odb->will_destroy) goto out; if (fsync_object_files > 0) - fsync_or_die(fd, "loose object file"); + fsync_or_die(fd, filename); else fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, - "loose object file"); + filename); out: if (close(fd) != 0) @@ -2011,7 +2011,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr, die(_("confused by unstable object source data for %s"), oid_to_hex(oid)); - close_loose_object(fd); + close_loose_object(fd, tmp_file.buf); if (mtime) { struct utimbuf utb; From patchwork Wed Mar 30 05:05:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A58B6C433EF for ; Wed, 30 Mar 2022 05:05:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242736AbiC3FHi (ORCPT ); Wed, 30 Mar 2022 01:07:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242717AbiC3FH2 (ORCPT ); Wed, 30 Mar 2022 01:07:28 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBB2A65D19 for ; Tue, 29 Mar 2022 22:05:41 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id l7-20020a05600c1d0700b0038c99618859so485529wms.2 for ; Tue, 29 Mar 2022 22:05:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QWCWFNWsBB6VcLQ3/PKqrQkuCpYYV6HLUH3YP6F+dLc=; b=Ff9hDDzMUJSPOPpB5tpOaCOkTNuKS+dqFe8ajAL2/isPutkb3vFdRycpmRBwpvNZA1 RdTl5MyP7DswHjRaYsiJGSOIYJodmllZ6Da+n0ydXCuGDCGTxtdorZyUZEZCIUSKlp7P p5gusGBaA7yxQ12pgORziFzJuNjSuvRkau+sA5ntZmvdJfy7MB5B2p4+rJxget9ZzBTB JmSxwNW5nIsZbyljc73EPID3mVMhsDXYnQK9jgJVjwwCLGs9RszdglKGRg47t826AmaH 33FwRPfTJIBJSFCNTtrVUYfm4kMwt7VceFL2BrE38GklQL3eJiipa8E3a/6H+wZmDRxn O1Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QWCWFNWsBB6VcLQ3/PKqrQkuCpYYV6HLUH3YP6F+dLc=; b=47sVxRUZzqIGNugxPxWXm72iFcFtBJ8HIA0oXAzeFn6FJt5y7yvZJyfKuj/AqOxgjM qPbBI2MFr2lTS1xOKh+hTR5CwUORqrssHCsa75iQwYumsFPtvtEdwJdzni9vIDb1EzCL 6SaKWnbQ9kzfJ6hZ2KJcj6SbOm/s/rCXzK66RCH8KUZmrJcEHSFHk9rm85OXS8UKgxeJ k9ATycnoPtexQuv9bgn+QRFOFfjnnN4BFrAgC1yjLUvcO7WYgWM1F5FRmx6K061S2jVA Zyz8v/W93iGYrNpLPpyx9/A0RxEEANtVxvgCnhIx91PnVAlviAV+8o/tThvR5QfAhTgw aH9w== X-Gm-Message-State: AOAM5336GaEa70qRT7FC5BlaKjWoXG+zHq8OoDh2KAdcRaYhy65dFPh/ c38BOsREnHpW+fs2cMtPD/BtZZ/zwAI= X-Google-Smtp-Source: ABdhPJz7T2oQft3mxegYpGaURAIBG2WyibOwuMMXvdvrLj6D7gJ0l0A2bdHwI3KtljcmaAED93HVuw== X-Received: by 2002:a7b:ce02:0:b0:381:2007:f75c with SMTP id m2-20020a7bce02000000b003812007f75cmr2698745wmc.6.1648616739854; Tue, 29 Mar 2022 22:05:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az19-20020a05600c601300b0038cadf3aa69sm5438063wmb.36.2022.03.29.22.05.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:39 -0700 (PDT) Message-Id: <9e7ae22fa4a2693fe26659f875dd780080c4cfb2.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:22 +0000 Subject: [PATCH v5 04/14] core.fsyncmethod: batched disk flushes for loose-objects Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh When adding many objects to a repo with `core.fsync=loose-object`, the cost of fsync'ing each object file can become prohibitive. One major source of the cost of fsync is the implied flush of the hardware writeback cache within the disk drive. This commit introduces a new `core.fsyncMethod=batch` option that batches up hardware flushes. It hooks into the bulk-checkin odb-transaction functionality, takes advantage of tmp-objdir, and uses the writeout-only support code. When the new mode is enabled, we do the following for each new object: 1a. Create the object in a tmp-objdir. 2a. Issue a pagecache writeback request and wait for it to complete. At the end of the entire transaction when unplugging bulk checkin: 1b. Issue an fsync against a dummy file to flush the log and hardware writeback cache, which should by now have seen the tmp-objdir writes. 2b. Rename all of the tmp-objdir files to their final names. 3b. When updating the index and/or refs, we assume that Git will issue another fsync internal to that operation. This is not the default today, but the user now has the option of syncing the index and there is a separate patch series to implement syncing of refs. On a filesystem with a singular journal that is updated during name operations (e.g. create, link, rename, etc), such as NTFS, HFS+, or XFS we would expect the fsync to trigger a journal writeout so that this sequence is enough to ensure that the user's data is durable by the time the git command returns. This sequence also ensures that no object files appear in the main object store unless they are fsync-durable. Batch mode is only enabled if core.fsync includes loose-objects. If the legacy core.fsyncObjectFiles setting is enabled, but core.fsync does not include loose-objects, we will use file-by-file fsyncing. In step (1a) of the sequence, the tmp-objdir is created lazily to avoid work if no loose objects are ever added to the ODB. We use a tmp-objdir to maintain the invariant that no loose-objects are visible in the main ODB unless they are properly fsync-durable. This is important since future ODB operations that try to create an object with specific contents will silently drop the new data if an object with the target hash exists without checking that the loose-object contents match the hash. Only a full git-fsck would restore the ODB to a functional state where dataloss doesn't occur. In step (1b) of the sequence, we issue a fsync against a dummy file created specifically for the purpose. This method has a little higher cost than using one of the input object files, but makes adding new callers of this mechanism easier, since we don't need to figure out which object file is "last" or risk sharing violations by caching the fd of the last object file. _Performance numbers_: Linux - Hyper-V VM running Kernel 5.11 (Ubuntu 20.04) on a fast SSD. Mac - macOS 11.5.1 running on a Mac mini on a 1TB Apple SSD. Windows - Same host as Linux, a preview version of Windows 11. Adding 500 files to the repo with 'git add' Times reported in seconds. object file syncing | Linux | Mac | Windows --------------------|-------|-------|-------- disabled | 0.06 | 0.35 | 0.61 fsync | 1.88 | 11.18 | 2.47 batch | 0.15 | 0.41 | 1.53 Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 8 ++++ bulk-checkin.c | 71 +++++++++++++++++++++++++++++++++++ bulk-checkin.h | 3 ++ cache.h | 8 +++- config.c | 2 + object-file.c | 7 +++- 6 files changed, 97 insertions(+), 2 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 9da3e5d88f6..3c90ba0b395 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -596,6 +596,14 @@ core.fsyncMethod:: * `writeout-only` issues pagecache writeback requests, but depending on the filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. +* `batch` enables a mode that uses writeout-only flushes to stage multiple + updates in the disk writeback cache and then does a single full fsync of + a dummy file to trigger the disk cache flush at the end of the operation. ++ + Currently `batch` mode only applies to loose-object files. Other repository + data is made durable as if `fsync` was specified. This mode is expected to + be as safe as `fsync` on macOS for repos stored on HFS+ or APFS filesystems + and on Windows for repos stored on NTFS or ReFS filesystems. core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. diff --git a/bulk-checkin.c b/bulk-checkin.c index 8b0fd5c7723..9799d247cad 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -3,15 +3,20 @@ */ #include "cache.h" #include "bulk-checkin.h" +#include "lockfile.h" #include "repository.h" #include "csum-file.h" #include "pack.h" #include "strbuf.h" +#include "string-list.h" +#include "tmp-objdir.h" #include "packfile.h" #include "object-store.h" static int odb_transaction_nesting; +static struct tmp_objdir *bulk_fsync_objdir; + static struct bulk_checkin_state { char *pack_tmp_name; struct hashfile *f; @@ -80,6 +85,40 @@ clear_exit: reprepare_packed_git(the_repository); } +/* + * Cleanup after batch-mode fsync_object_files. + */ +static void do_batch_fsync(void) +{ + struct strbuf temp_path = STRBUF_INIT; + struct tempfile *temp; + + if (!bulk_fsync_objdir) + return; + + /* + * Issue a full hardware flush against a temporary file to ensure + * that all objects are durable before any renames occur. The code in + * fsync_loose_object_bulk_checkin has already issued a writeout + * request, but it has not flushed any writeback cache in the storage + * hardware or any filesystem logs. This fsync call acts as a barrier + * to ensure that the data in each new object file is durable before + * the final name is visible. + */ + strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX", get_object_directory()); + temp = xmks_tempfile(temp_path.buf); + fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp)); + delete_tempfile(&temp); + strbuf_release(&temp_path); + + /* + * Make the object files visible in the primary ODB after their data is + * fully durable. + */ + tmp_objdir_migrate(bulk_fsync_objdir); + bulk_fsync_objdir = NULL; +} + static int already_written(struct bulk_checkin_state *state, struct object_id *oid) { int i; @@ -274,6 +313,36 @@ static int deflate_to_pack(struct bulk_checkin_state *state, return 0; } +void prepare_loose_object_bulk_checkin(void) +{ + /* + * We lazily create the temporary object directory + * the first time an object might be added, since + * callers may not know whether any objects will be + * added at the time they call begin_odb_transaction. + */ + if (!odb_transaction_nesting || bulk_fsync_objdir) + return; + + bulk_fsync_objdir = tmp_objdir_create("bulk-fsync"); + if (bulk_fsync_objdir) + tmp_objdir_replace_primary_odb(bulk_fsync_objdir, 0); +} + +void fsync_loose_object_bulk_checkin(int fd, const char *filename) +{ + /* + * If we have an active ODB transaction, we issue a call that + * cleans the filesystem page cache but avoids a hardware flush + * command. Later on we will issue a single hardware flush + * before as part of do_batch_fsync. + */ + if (!bulk_fsync_objdir || + git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) { + fsync_or_die(fd, filename); + } +} + int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags) @@ -301,4 +370,6 @@ void end_odb_transaction(void) if (bulk_checkin_state.f) finish_bulk_checkin(&bulk_checkin_state); + + do_batch_fsync(); } diff --git a/bulk-checkin.h b/bulk-checkin.h index 69a94422ac7..70edf745be8 100644 --- a/bulk-checkin.h +++ b/bulk-checkin.h @@ -6,6 +6,9 @@ #include "cache.h" +void prepare_loose_object_bulk_checkin(void); +void fsync_loose_object_bulk_checkin(int fd, const char *filename); + int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags); diff --git a/cache.h b/cache.h index ef7d34b7a09..a5bf15a5131 100644 --- a/cache.h +++ b/cache.h @@ -1040,7 +1040,8 @@ extern int use_fsync; enum fsync_method { FSYNC_METHOD_FSYNC, - FSYNC_METHOD_WRITEOUT_ONLY + FSYNC_METHOD_WRITEOUT_ONLY, + FSYNC_METHOD_BATCH, }; extern enum fsync_method fsync_method; @@ -1767,6 +1768,11 @@ void fsync_or_die(int fd, const char *); int fsync_component(enum fsync_component component, int fd); void fsync_component_or_die(enum fsync_component component, int fd, const char *msg); +static inline int batch_fsync_enabled(enum fsync_component component) +{ + return (fsync_components & component) && (fsync_method == FSYNC_METHOD_BATCH); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/config.c b/config.c index 3c9b6b589ab..511f4584eeb 100644 --- a/config.c +++ b/config.c @@ -1688,6 +1688,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb) fsync_method = FSYNC_METHOD_FSYNC; else if (!strcmp(value, "writeout-only")) fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else if (!strcmp(value, "batch")) + fsync_method = FSYNC_METHOD_BATCH; else warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); diff --git a/object-file.c b/object-file.c index 5ffbf3d4fd4..d2e0c13198f 100644 --- a/object-file.c +++ b/object-file.c @@ -1893,7 +1893,9 @@ static void close_loose_object(int fd, const char *filename) if (the_repository->objects->odb->will_destroy) goto out; - if (fsync_object_files > 0) + if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT)) + fsync_loose_object_bulk_checkin(fd, filename); + else if (fsync_object_files > 0) fsync_or_die(fd, filename); else fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, @@ -1961,6 +1963,9 @@ static int write_loose_object(const struct object_id *oid, char *hdr, static struct strbuf tmp_file = STRBUF_INIT; static struct strbuf filename = STRBUF_INIT; + if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT)) + prepare_loose_object_bulk_checkin(); + loose_object_path(the_repository, &filename, oid); fd = create_tmpfile(&tmp_file, filename.buf); From patchwork Wed Mar 30 05:05:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 204E7C433F5 for ; Wed, 30 Mar 2022 05:05:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242739AbiC3FHj (ORCPT ); Wed, 30 Mar 2022 01:07:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242718AbiC3FH2 (ORCPT ); Wed, 30 Mar 2022 01:07:28 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A220D6621D for ; Tue, 29 Mar 2022 22:05:42 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id u3so27596918wrg.3 for ; Tue, 29 Mar 2022 22:05:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0FZmb8xpqwGRKdPb/QtLOYQHy803cgaUCogAov31nBg=; b=odu0pFnB2zMMqIb6Ll98sDstvzgLsz7FI34OjFDE3O+t0QGwqgOq8xfFe9jcEQsU37 wGtUIuLq9IXYj7N7vHC9TOkY1hL7Hsuny4YvsPWNQPauVTnA05zVDc+5dHgCJnE4kT6i gmGQOmvh3v7ihMCQglP/28HT5c/i9UBCcjVbmkBusXKVowbXnIpVim+hutjLGUJHKS4E MeyhdwDrOKc0ZfdLvJLLkkzcKm58F9bZQEGXTudBsGDi3Go/Tsa/kzX+ZwuaI6cR+Ihk HCqxc/w0JPBjleDeZq+rgcrItPvQfapZ1ojLGIe5qXtgFpVCcuZFx2qe1UEa6uxzOpO0 EoGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0FZmb8xpqwGRKdPb/QtLOYQHy803cgaUCogAov31nBg=; b=UWo8Mmg+Qx1ZnDNtXeOCK3+k8+bUeQshnMqmklVc59zfq2mQRS169q2WVIhZ/tcqRQ +pKTUAMQBAK8T8hs9HsgR8EkdAGwoaNyAHPBhhT1413Nok33cp4OszTsElNrqWwklF5p zBQXKVH+AgfLtrgkPC2LNct/bL7Fct5cd5+Ysee/JMQh1otuiTW7FxEFPWtpEDQoq3f5 8tUjeV35giY1g0LE4Wm5FOBIYkQHYWlhEevcTU0e5vP6SOFI9SqqeM29TktmLGySY/1w aS+A7xHoXtR3JCLtnnlmBK0OWkV5cBwBfDotrIZDcahXuP0h7eCysrNAVTjylDQY7iAJ kyyg== X-Gm-Message-State: AOAM532FoXr4x8326EO5VQSv9xWsdk/H+5sWaAXhr84UYpVX/o1q2e6Y b6nJkOMvzn7g5S7C0XFJoNRLqTeYcm4= X-Google-Smtp-Source: ABdhPJzE6jTAIeor/HhpUsV356yXu4m8g2QYcfEttZ820n3IGT1PMzKYYs6fORzYsu/wxFO6A5WbHw== X-Received: by 2002:a5d:47a1:0:b0:204:9f5:e72f with SMTP id 1-20020a5d47a1000000b0020409f5e72fmr36157650wrb.656.1648616740945; Tue, 29 Mar 2022 22:05:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e9-20020a5d6d09000000b00203ecdca5b7sm17585773wrq.33.2022.03.29.22.05.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:40 -0700 (PDT) Message-Id: <83fa4a5f3a5c79fa814932c0705867ff16a584c7.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:23 +0000 Subject: [PATCH v5 05/14] cache-tree: use ODB transaction around writing a tree Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Take advantage of the odb transaction infrastructure around writing the cached tree to the object database. Signed-off-by: Neeraj Singh --- cache-tree.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cache-tree.c b/cache-tree.c index 6752f69d515..8c5e8822716 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -3,6 +3,7 @@ #include "tree.h" #include "tree-walk.h" #include "cache-tree.h" +#include "bulk-checkin.h" #include "object-store.h" #include "replace-object.h" #include "promisor-remote.h" @@ -474,8 +475,10 @@ int cache_tree_update(struct index_state *istate, int flags) trace_performance_enter(); trace2_region_enter("cache_tree", "update", the_repository); + begin_odb_transaction(); i = update_one(istate->cache_tree, istate->cache, istate->cache_nr, "", 0, &skip, flags); + end_odb_transaction(); trace2_region_leave("cache_tree", "update", the_repository); trace_performance_leave("cache_tree_update"); if (i < 0) From patchwork Wed Mar 30 05:05:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59F5FC433EF for ; Wed, 30 Mar 2022 05:06:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242740AbiC3FHz (ORCPT ); Wed, 30 Mar 2022 01:07:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242720AbiC3FH2 (ORCPT ); Wed, 30 Mar 2022 01:07:28 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CAF5266623 for ; Tue, 29 Mar 2022 22:05:43 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id u3so27596951wrg.3 for ; Tue, 29 Mar 2022 22:05:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4E3crma7zgCOi3ds7U5p/TJuGKDkbTL9Fu3rj9gxZZo=; b=l2PdaytBMynmT5Baxm+MKxxBN2g+OAWR1bHuB5m0ddDUQpTGARJ2Lne1sTEjcCRHCF Plaa2TFUoZUvEiiQVBMHZ3GINnI83Ld+ClahjDHXXKRi+mfIvus22D0qF0WiameGWduh RoB+Y/SvXTJcr8yOCaNnx4eM7QWXaS1fvDWr5gOnpqjO9iP68AUDGb2I5YLzuBa3HIX3 79YLVp8UHT8IlHcMBLdxdIEh3djCJRMVXLeVXCMV1Hpacsd0mF8+yqstIap+jTizyFBv HMbUb320B6ISuNcYIfmxls88XFvqOUAK3uNEU0jiKqyY5QO++G3qoflIUgx3wRuTq6XS EirQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4E3crma7zgCOi3ds7U5p/TJuGKDkbTL9Fu3rj9gxZZo=; b=ue0TT7lsCAlsmJedHnSZR3UomS6bkxczIh6c3DYGM45GEeqtwVq8NYmHnKEqqKDyoZ 9AyJE2yURIuwIzihAWVxxd8n6ec64EQ/9nUgaZkpyFecnG6T8/32UctzCY2a3c0LwiYX RC6gcCHGozCfallh5iJ4ooxW9kPnbILwaZUn9Mr7e/8evLj0F/pHa5ijn9JhD7Ty2MNN fsLxmELMdpgTDUfv/TEckUaX1Z38cSXtkqMl3fWkcqc0fI5i4dqudFDkAaCaCMvtgXVq xr7Iu67/2h3BcDB1crHU6hvKBK880UTe2a+sQSxi0WaBx69bXRzxIQ+yH33ItTxAqVHe eRQA== X-Gm-Message-State: AOAM532txWh/+sQG950kgvwa3LJp6HOZzeNJCAxuzWnx/y8hLAcG7q4l Q2F9gtaGBm4MLtI/yLS/bfP8nK31q1E= X-Google-Smtp-Source: ABdhPJzwHAnguen1JiGdUzmUXnAaGii/vFyOi/M0DiwQyvt2zUf6mOrfY8CSJpgZDepJ7s3WjOBShw== X-Received: by 2002:adf:d1c9:0:b0:203:c225:d042 with SMTP id b9-20020adfd1c9000000b00203c225d042mr34036424wrd.630.1648616741795; Tue, 29 Mar 2022 22:05:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e14-20020a5d65ce000000b00205c0cb33e0sm7417769wrw.35.2022.03.29.22.05.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:41 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:24 +0000 Subject: [PATCH v5 06/14] builtin/add: add ODB transaction around add_files_to_cache Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The add_files_to_cache function is invoked internally by builtin/commit.c and builtin/checkout.c for their flags that stage modified files before doing the larger operation. These commands can benefit from batched fsyncing. Signed-off-by: Neeraj Singh --- builtin/add.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/builtin/add.c b/builtin/add.c index 9bf37ceae8e..e39770e4746 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -141,7 +141,16 @@ int add_files_to_cache(const char *prefix, rev.diffopt.format_callback_data = &data; rev.diffopt.flags.override_submodule_config = 1; rev.max_count = 0; /* do not compare unmerged paths with stage #2 */ + + /* + * Use an ODB transaction to optimize adding multiple objects. + * This function is invoked from commands other than 'add', which + * may not have their own transaction active. + */ + begin_odb_transaction(); run_diff_files(&rev, DIFF_RACY_IS_MODIFIED); + end_odb_transaction(); + clear_pathspec(&rev.prune_data); return !!data.add_errors; } From patchwork Wed Mar 30 05:05:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CB86C433F5 for ; Wed, 30 Mar 2022 05:06:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242760AbiC3FHt (ORCPT ); Wed, 30 Mar 2022 01:07:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242740AbiC3FHp (ORCPT ); Wed, 30 Mar 2022 01:07:45 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B991D66AFE for ; Tue, 29 Mar 2022 22:05:44 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id q20so11523434wmq.1 for ; Tue, 29 Mar 2022 22:05:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RIF5CAZ9hTXQxe5JpOeXS1KOLlyZ8DHo3e5ICWJYFAY=; b=cg/VnRJVSzjavnMvPA0/sBzx3C/xMKQWwbSgHtMMsqjAXAMVj0iAk6bzo2J4ceFEnZ DrAPIrHQ81d0M7khoYbxhILjvHeayiWYLSCi5a213L7gBPHkz79DRDO3kGq9f7gM8Cui ufQksd8rSokkcQrCRzcPoXZyBTilLMlUGr7qiWGepH4Xp/7mCW7C/Cb3Doqvb8RK9CaR 300wM+q82YZmAyW0nivvLAdbP2W2AO7UxQHIti7WxSlRvvFIbRlx5cPYaj7UUKctfnU8 kX49HcwkQckCJWHbFo5FKeBMf8hi4t7zowy/9gzRvMqQjg3CRRpeBpqpmcE7tFsK1xuS kNbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RIF5CAZ9hTXQxe5JpOeXS1KOLlyZ8DHo3e5ICWJYFAY=; b=nbJLPNXFGlqUk+12bESmEBn0yetstvt9oa6riHklY7WjBegDBwJH70vwOCNR/eC0HT dyH4qEd+GfKKXcQsKA0vjs7ujUXsHMTB1QiTEohOe2WFHmWfcmhzWG3r+ABBpc7U6gaf x5cGd9RHoOplcBzOgd/rfsAtL+EtgdFznRJsSoX8qMsHzZ9tJVezwIRfkxnCFZFVA/gN Mt1pq/yBlQGGQpJZyh7HaDOk5u9WtVi2Jcla1qr6zYlbQp9RFCu+27HehuPhVb9LHXJy UErDoAyCIwGUAC+OuYD0dC/qhLYVzmAye16MkJtfNQ3T22Gw9MBpq3zuhR3L3UguynLa 5b7g== X-Gm-Message-State: AOAM533b8Isf3PDEXLlPpIeLYPAFAO2pCe0n72vt8wWjIS/dpS87z8yh IWdZ5e4KXafEmaxJaO49lxKTk079SO4= X-Google-Smtp-Source: ABdhPJzfLqDSyjhxsdal6OCNEUE3nZ8W8MH/83Gq7TzZzOemiQzczj+7ByUUvwtKpmicQwSld9eX1Q== X-Received: by 2002:a1c:7219:0:b0:38c:a4f8:484f with SMTP id n25-20020a1c7219000000b0038ca4f8484fmr2537303wmc.99.1648616742893; Tue, 29 Mar 2022 22:05:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q17-20020adff951000000b00205c1b97ac4sm6690967wrr.20.2022.03.29.22.05.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:42 -0700 (PDT) Message-Id: <8cac94598a58704d9b625a9d8a593779f7adc30f.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:25 +0000 Subject: [PATCH v5 07/14] update-index: use the bulk-checkin infrastructure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The update-index functionality is used internally by 'git stash push' to setup the internal stashed commit. This change enables odb-transactions for update-index infrastructure to speed up adding new objects to the object database by leveraging the batch fsync functionality. There is some risk with this change, since under batch fsync, the object files will be in a tmp-objdir until update-index is complete, so callers using the --stdin option will not see them until update-index is done. This risk is mitigated by not keeping an ODB transaction open around --stdin processing if in --verbose mode. Without --verbose mode, a caller feeding update-index via --stdin wouldn't know when update-index adds an object, event without an ODB transaction. Signed-off-by: Neeraj Singh --- builtin/update-index.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/builtin/update-index.c b/builtin/update-index.c index aafe7eeac2a..50f9063e1c6 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -5,6 +5,7 @@ */ #define USE_THE_INDEX_COMPATIBILITY_MACROS #include "cache.h" +#include "bulk-checkin.h" #include "config.h" #include "lockfile.h" #include "quote.h" @@ -1116,6 +1117,12 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) */ parse_options_start(&ctx, argc, argv, prefix, options, PARSE_OPT_STOP_AT_NON_OPTION); + + /* + * Allow the object layer to optimize adding multiple objects in + * a batch. + */ + begin_odb_transaction(); while (ctx.argc) { if (parseopt_state != PARSE_OPT_DONE) parseopt_state = parse_options_step(&ctx, options, @@ -1167,6 +1174,17 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) the_index.version = preferred_index_format; } + /* + * It is possible, though unlikely, that a caller could use the verbose + * output to synchronize with addition of objects to the object + * database. The current implementation of ODB transactions leaves + * objects invisible while a transaction is active, so end the + * transaction here if verbose output is enabled. + */ + + if (verbose) + end_odb_transaction(); + if (read_from_stdin) { struct strbuf buf = STRBUF_INIT; struct strbuf unquoted = STRBUF_INIT; @@ -1190,6 +1208,12 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) strbuf_release(&buf); } + /* + * By now we have added all of the new objects + */ + if (!verbose) + end_odb_transaction(); + if (split_index > 0) { if (git_config_get_split_index() == 0) warning(_("core.splitIndex is set to false; " From patchwork Wed Mar 30 05:05:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1DE9C4332F for ; Wed, 30 Mar 2022 05:06:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242729AbiC3FHy (ORCPT ); Wed, 30 Mar 2022 01:07:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242718AbiC3FHp (ORCPT ); Wed, 30 Mar 2022 01:07:45 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCC4D66C81 for ; Tue, 29 Mar 2022 22:05:45 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id w4so27527331wrg.12 for ; Tue, 29 Mar 2022 22:05:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5ebJFCI4F0cWCTTRJ8U340O4g9iVrZ+6k6F3XALLE/c=; b=bGyhwMSlcIM3pgO/RPkKirETu2KgJ5/z7bLsZlG9CQQ2EK360HFD8rWPV+jzH4u41q nPpHInkp0MAhLxpJXTVofWXLN4+G6VwDUs9F6oHlp/O+AlKAz5edaH3+UWVR0vfk+PG0 +FKCEwzRQXZSVkSu9l0VpQvXEaGfrACl2J5SpIi9jSW3ijayoxq4sfhVd2/DkiJuzZz2 yrQvgKLfQFJ04Izrrlx3Rwkn+CBxlztgrJhT49MCzuEKzim1Kh8tqan6LDV8Su4/vqtD NR6yE3UxptYUW0hmQXmRkOeoNoLikv5LoQhPdWTSVSQtvPBd7MLRUV7xlc0d25+M+x2u zivw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5ebJFCI4F0cWCTTRJ8U340O4g9iVrZ+6k6F3XALLE/c=; b=YGEqks31K8rD7T67RV6P4PktTCuCievB71gj9qgoCuTe5o8FGvmwJJ8WP8Yr7BYL4o Oa3yzuTzfePE1OJYkFy9THpvhmScVVDuAJGHwf4Pr8eGTd3whdaERKjA0hhmPHIdveqw +7KpbOaidwq+5DjHnyocTpVvBNIrXdU3PkSN2rfHkB5LOq36jSkKPKDhvqXyEKJeydLS tAMuhMfmh+IcbKG6wm8FhBlvXZoexZLwLDJAICqmrjiPxSYr7zypFlIHdpPYuP6/t6N6 LQ3VvTtEW4Q77v9WtKTva/TmdOJGiyZdYVoc3CY6iwEOzAfB18L9g5XE3Yute6jrxxEd njSg== X-Gm-Message-State: AOAM530d1IQtm1V3dwEiWRx3Z5e83HjeK3Da4LTd5HmjWD3OyKOi3r2Z v0JJTziSl0QeC9+4FS6CVmvoHh6C+34= X-Google-Smtp-Source: ABdhPJw7UZMOQbSoNIWLMDPk6MyyLjgRrWHJx5ch4NkqhP9GO/kbZV3kCBTHiAfjBDU9K3yrRzxcPg== X-Received: by 2002:a05:6000:1704:b0:203:d857:aa7a with SMTP id n4-20020a056000170400b00203d857aa7amr33449963wrc.513.1648616743983; Tue, 29 Mar 2022 22:05:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q16-20020adff950000000b00205aa05fa03sm12166221wrr.58.2022.03.29.22.05.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:43 -0700 (PDT) Message-Id: <523e5fbd63ef85035131bd4cec7565707c290e84.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:26 +0000 Subject: [PATCH v5 08/14] unpack-objects: use the bulk-checkin infrastructure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The unpack-objects functionality is used by fetch, push, and fast-import to turn the transfered data into object database entries when there are fewer objects than the 'unpacklimit' setting. By enabling an odb-transaction when unpacking objects, we can take advantage of batched fsyncs. Here are some performance numbers to justify batch mode for unpack-objects, collected on a WSL2 Ubuntu VM. Fsync Mode | Time for 90 objects (ms) ------------------------------------- Off | 170 On,fsync | 760 On,batch | 230 Note that the default unpackLimit is 100 objects, so there's a 3x benefit in the worst case. The non-batch mode fsync scales linearly with the number of objects, so there are significant benefits even with smaller numbers of objects. Signed-off-by: Neeraj Singh --- builtin/unpack-objects.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c index dbeb0680a58..56d05e2725d 100644 --- a/builtin/unpack-objects.c +++ b/builtin/unpack-objects.c @@ -1,5 +1,6 @@ #include "builtin.h" #include "cache.h" +#include "bulk-checkin.h" #include "config.h" #include "object-store.h" #include "object.h" @@ -503,10 +504,12 @@ static void unpack_all(void) if (!quiet) progress = start_progress(_("Unpacking objects"), nr_objects); CALLOC_ARRAY(obj_list, nr_objects); + begin_odb_transaction(); for (i = 0; i < nr_objects; i++) { unpack_one(i); display_progress(progress, i + 1); } + end_odb_transaction(); stop_progress(&progress); if (delta_list) From patchwork Wed Mar 30 05:05:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D1A3C433EF for ; Wed, 30 Mar 2022 05:06:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242798AbiC3FHt (ORCPT ); Wed, 30 Mar 2022 01:07:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242746AbiC3FHp (ORCPT ); Wed, 30 Mar 2022 01:07:45 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FFB96622D for ; Tue, 29 Mar 2022 22:05:46 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id u16so27568858wru.4 for ; Tue, 29 Mar 2022 22:05:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=hg8plsugjefXxhMhSvzABM0c1aAu2C8TxytBC4+PwWI=; b=QizEu2+HroFKYp+23Jz5+0r/DIb/JSLKxwAgToyHL1YJcRuOwexmVweZrUqIEsa3aX I4Ou1CXWxLIZ44/73/idzKMjnzQo3jjKorOHThCfVpJlsBC0tTwv1W6VnDv/cfqY+3q2 ELUgxtkksHjtNShpza7kjqtcwKVIBJHqXFQzDiVsBVzYgJde5VI5Q08aL4UnSTCbXOih qBefJY+qFwLSgK4L9Eh74lrwkK8yCKZQtGaXwwaZv1yPZiX3l7/2Jx8rXGv5Opdo+Jg+ 3OWGjFyQXro7lJDbxYe+AgRLhUjrCJ6KaoOL2kZIal7hdCo8mnD3Y058bLJT5i3JSH4O Qy4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=hg8plsugjefXxhMhSvzABM0c1aAu2C8TxytBC4+PwWI=; b=AjUwKEoAQ2k4Sw+Ec/IaJbA+O1/VAWjVbS8iFPTKf/jfRyQM7ftgU7KgCPudn1hxl2 nkO4jyvzH6Jg1w37kp13j9/wAyzJwTl5smwZ+M2F276hNGJe4tAjASnWruFnvidbt24P EhEwJOPp3YM4/EF3nnlth3lmnoyVN/OwyHd1gLwp1OaHJHrVrZpK1wXpH+JTuX3Lw4n6 BRr8EKV+BwITnqrJa++uZWIfm4idXDYwVKiOP7xikaDWrv32Yqg+1lXm7HJVpglbzmI+ ql7V9X551Aww/nOa30St5udw3Q1gW3HV5Rv5PUwo5VSzFXeVhRtOZ0V6nzAJjJ3fZZrC uaow== X-Gm-Message-State: AOAM532V8kMrCm1se7HbtySwo/O81XYf7COPzoD13RF8djYy3rpaJpCt cCiKyx9aFU6Fd2uU72Ji2KJPlH1hOLE= X-Google-Smtp-Source: ABdhPJxGPGp28QmmEhC3/p8Iw7UoNzjvM0UA3pTFfd7HtZIPmY1lWWVw1MbMygjeSt5OEmGaESi/Ng== X-Received: by 2002:adf:da50:0:b0:204:7d8:b654 with SMTP id r16-20020adfda50000000b0020407d8b654mr35044691wrl.607.1648616744824; Tue, 29 Mar 2022 22:05:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y5-20020a1c4b05000000b0038cbf571334sm3578152wma.18.2022.03.29.22.05.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:44 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:27 +0000 Subject: [PATCH v5 09/14] core.fsync: use batch mode and sync loose objects by default on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Git for Windows has defaulted to core.fsyncObjectFiles=true since September 2017. We turn on syncing of loose object files with batch mode in upstream Git so that we can get broad coverage of the new code upstream. We don't actually do fsyncs in the most of the test suite, since GIT_TEST_FSYNC is set to 0. However, we do exercise all of the surrounding batch mode code since GIT_TEST_FSYNC merely makes the maybe_fsync wrapper always appear to succeed. Signed-off-by: Neeraj Singh --- cache.h | 4 ++++ compat/mingw.h | 3 +++ config.c | 2 +- git-compat-util.h | 2 ++ 4 files changed, 10 insertions(+), 1 deletion(-) diff --git a/cache.h b/cache.h index a5bf15a5131..7f6cbb254b4 100644 --- a/cache.h +++ b/cache.h @@ -1031,6 +1031,10 @@ enum fsync_component { FSYNC_COMPONENT_INDEX | \ FSYNC_COMPONENT_REFERENCE) +#ifndef FSYNC_COMPONENTS_PLATFORM_DEFAULT +#define FSYNC_COMPONENTS_PLATFORM_DEFAULT FSYNC_COMPONENTS_DEFAULT +#endif + /* * A bitmask indicating which components of the repo should be fsynced. */ diff --git a/compat/mingw.h b/compat/mingw.h index 6074a3d3ced..afe30868c04 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -332,6 +332,9 @@ int mingw_getpagesize(void); int win32_fsync_no_flush(int fd); #define fsync_no_flush win32_fsync_no_flush +#define FSYNC_COMPONENTS_PLATFORM_DEFAULT (FSYNC_COMPONENTS_DEFAULT | FSYNC_COMPONENT_LOOSE_OBJECT) +#define FSYNC_METHOD_DEFAULT (FSYNC_METHOD_BATCH) + struct rlimit { unsigned int rlim_cur; }; diff --git a/config.c b/config.c index 511f4584eeb..e9cac5f4707 100644 --- a/config.c +++ b/config.c @@ -1342,7 +1342,7 @@ static const struct fsync_component_name { static enum fsync_component parse_fsync_components(const char *var, const char *string) { - enum fsync_component current = FSYNC_COMPONENTS_DEFAULT; + enum fsync_component current = FSYNC_COMPONENTS_PLATFORM_DEFAULT; enum fsync_component positive = 0, negative = 0; while (string) { diff --git a/git-compat-util.h b/git-compat-util.h index 0892e209a2f..fffe42ce7c1 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1257,11 +1257,13 @@ __attribute__((format (printf, 3, 4))) NORETURN void BUG_fl(const char *file, int line, const char *fmt, ...); #define BUG(...) BUG_fl(__FILE__, __LINE__, __VA_ARGS__) +#ifndef FSYNC_METHOD_DEFAULT #ifdef __APPLE__ #define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY #else #define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC #endif +#endif enum fsync_action { FSYNC_WRITEOUT_ONLY, From patchwork Wed Mar 30 05:05:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A93D5C433F5 for ; Wed, 30 Mar 2022 05:06:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242741AbiC3FHw (ORCPT ); Wed, 30 Mar 2022 01:07:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242753AbiC3FHq (ORCPT ); Wed, 30 Mar 2022 01:07:46 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FC3F66213 for ; Tue, 29 Mar 2022 22:05:47 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id 123-20020a1c1981000000b0038b3616a71aso2704973wmz.4 for ; Tue, 29 Mar 2022 22:05:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=f/ZbN5quD6ggV2+4j9lRbjM6zktg3vb1d35PK4/3tT4=; b=CRxoiIULfFKgVM5bIjkDlyfXOsp56IVxfY4TNnUPuG7X9w05jYMK16rUQKGIbhnH5i AroW7RAi+g86v0CAYG7cpFWkScoLelDZWJq2R7QTaEFPhRqXT5ZQ/G8XaaG0/v0Ix3PJ cZZ2gMcEESxVXQruJMmixBcP3/M4riIOks99Of5P+dG9sLS1ydCBea1UJIPlb14twi3q Ok33EKIeCsEhUTEJethtHpYXcSG9SEzus7ZNmnUsOQPET/+tQEcACKmjEcVLWGIvZF4N Ct5RxqTyoHVpReGD61X1mUsBjm/KYODsYVSnuAGlL3JeWUfMCNeu0ZaAqhqfaLwYRLkE ylrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=f/ZbN5quD6ggV2+4j9lRbjM6zktg3vb1d35PK4/3tT4=; b=dyeNCXR8fI2ys5rOAC3IPFrhrS1g9HGEfOq3WAcgkfJU1I2Ujyrb1GtvFQZF2z+J3y X+CT5vT82d0guufSEH0IB2r6DAOn4WLFBWsjmAM+Iy71Ed4cODP6cR7bkdElFWLF/PXJ ueU9qfRrdFmhlejEV+RsORRFib0/KORQIwzzS/7p4gble44HaV1L9yei4wTyvQAcZfIa GPvuu9NYswaFJcs90FIU7V1Jg/bXUwi/qiQBgOgixCKm7KTWbQKopjkHlO0BpykvCy/k kFjx75+l9Gu2hsDpiFdPvaSquldh0kcBOOd17hnQ+k1LwTC8vzdSUuue6kFKFOA5OHxu 9Qaw== X-Gm-Message-State: AOAM532wvGg6DnfQuZYZ/xO+Xuv4kAvriVB2bRHbhYWcfgQ+9fup8Gj3 7vKmMVS2YOkByqzpoUZpdmXuyekkZxU= X-Google-Smtp-Source: ABdhPJwOTloCXdp/x2NkpadXrEOUnn7lEsRT1+pwTIcxHWftVAFOIsYKQL4LivgxWKQkABLWZou/Mw== X-Received: by 2002:a7b:cf03:0:b0:38e:38c0:be27 with SMTP id l3-20020a7bcf03000000b0038e38c0be27mr141760wmg.150.1648616745693; Tue, 29 Mar 2022 22:05:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v14-20020a7bcb4e000000b0034492fa24c6sm3730371wmj.34.2022.03.29.22.05.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:45 -0700 (PDT) Message-Id: <4de7300a7b0061c1399738c66ce05bfbbe2db1d0.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:28 +0000 Subject: [PATCH v5 10/14] test-lib-functions: add parsing helpers for ls-files and ls-tree Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Several tests use awk to parse OIDs from the output of 'git ls-files --stage' and 'git ls-tree'. Introduce helpers to centralize these uses of awk. Update t5317-pack-objects-filter-objects.sh to use the new ls-files helper so that it has some usages to review. Other updates are left for the future. Signed-off-by: Neeraj Singh --- t/t5317-pack-objects-filter-objects.sh | 91 +++++++++++++------------- t/test-lib-functions.sh | 10 +++ 2 files changed, 54 insertions(+), 47 deletions(-) diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 33b740ce628..bb633c9b099 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -10,9 +10,6 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME # Test blob:none filter. test_expect_success 'setup r1' ' - echo "{print \$1}" >print_1.awk && - echo "{print \$2}" >print_2.awk && - git init r1 && for n in 1 2 3 4 5 do @@ -22,10 +19,13 @@ test_expect_success 'setup r1' ' done ' +parse_verify_pack_blob_oid () { + awk '{print $1}' - +} + test_expect_success 'verify blob count in normal packfile' ' - git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 | + test_parse_ls_files_stage_oids | sort >expected && git -C r1 pack-objects --revs --stdout >all.pack <<-EOF && @@ -35,7 +35,7 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r1 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -54,12 +54,12 @@ test_expect_success 'verify blob:none packfile has no blobs' ' test_expect_success 'verify normal and blob:none packfiles have same commits/trees' ' git -C r1 verify-pack -v ../all.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >expected && git -C r1 verify-pack -v ../filter.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -123,8 +123,8 @@ test_expect_success 'setup r2' ' ' test_expect_success 'verify blob count in normal packfile' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout >all.pack <<-EOF && @@ -134,7 +134,7 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r2 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -161,8 +161,8 @@ test_expect_success 'verify blob:limit=1000' ' ' test_expect_success 'verify blob:limit=1001' ' - git -C r2 ls-files -s large.1000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=1001 >filter.pack <<-EOF && @@ -172,15 +172,15 @@ test_expect_success 'verify blob:limit=1001' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify blob:limit=10001' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=10001 >filter.pack <<-EOF && @@ -190,15 +190,15 @@ test_expect_success 'verify blob:limit=10001' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify blob:limit=1k' ' - git -C r2 ls-files -s large.1000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=1k >filter.pack <<-EOF && @@ -208,15 +208,15 @@ test_expect_success 'verify blob:limit=1k' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify explicitly specifying oversized blob in input' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && echo HEAD >objects && @@ -226,15 +226,15 @@ test_expect_success 'verify explicitly specifying oversized blob in input' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify blob:limit=1m' ' - git -C r2 ls-files -s large.1000 large.10000 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r2 ls-files -s large.1000 large.10000 | + test_parse_ls_files_stage_oids | sort >expected && git -C r2 pack-objects --revs --stdout --filter=blob:limit=1m >filter.pack <<-EOF && @@ -244,7 +244,7 @@ test_expect_success 'verify blob:limit=1m' ' git -C r2 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -253,12 +253,12 @@ test_expect_success 'verify blob:limit=1m' ' test_expect_success 'verify normal and blob:limit packfiles have same commits/trees' ' git -C r2 verify-pack -v ../all.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >expected && git -C r2 verify-pack -v ../filter.pack >verify_result && grep -E "commit|tree" verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -289,9 +289,8 @@ test_expect_success 'setup r3' ' ' test_expect_success 'verify blob count in normal packfile' ' - git -C r3 ls-files -s sparse1 sparse2 dir1/sparse1 dir1/sparse2 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r3 ls-files -s sparse1 sparse2 dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r3 pack-objects --revs --stdout >all.pack <<-EOF && @@ -301,7 +300,7 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r3 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -342,9 +341,8 @@ test_expect_success 'setup r4' ' ' test_expect_success 'verify blob count in normal packfile' ' - git -C r4 ls-files -s pattern sparse1 sparse2 dir1/sparse1 dir1/sparse2 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r4 ls-files -s pattern sparse1 sparse2 dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r4 pack-objects --revs --stdout >all.pack <<-EOF && @@ -354,19 +352,19 @@ test_expect_success 'verify blob count in normal packfile' ' git -C r4 verify-pack -v ../all.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify sparse:oid=OID' ' - git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r4 ls-files -s pattern >staged && - oid=$(awk -f print_2.awk staged) && + oid=$(test_parse_ls_files_stage_oids filter.pack <<-EOF && HEAD EOF @@ -374,15 +372,15 @@ test_expect_success 'verify sparse:oid=OID' ' git -C r4 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed ' test_expect_success 'verify sparse:oid=oid-ish' ' - git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r4 ls-files -s dir1/sparse1 dir1/sparse2 | + test_parse_ls_files_stage_oids | sort >expected && git -C r4 pack-objects --revs --stdout --filter=sparse:oid=main:pattern >filter.pack <<-EOF && @@ -392,7 +390,7 @@ test_expect_success 'verify sparse:oid=oid-ish' ' git -C r4 verify-pack -v ../filter.pack >verify_result && grep blob verify_result | - awk -f print_1.awk | + parse_verify_pack_blob_oid | sort >observed && test_cmp expected observed @@ -402,9 +400,8 @@ test_expect_success 'verify sparse:oid=oid-ish' ' # This models previously omitted objects that we did not receive. test_expect_success 'setup r1 - delete loose blobs' ' - git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 \ - >ls_files_result && - awk -f print_2.awk ls_files_result | + git -C r1 ls-files -s file.1 file.2 file.3 file.4 file.5 | + test_parse_ls_files_stage_oids | sort >expected && for id in `cat expected | sed "s|..|&/|"` diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index a027f0c409e..e6011409e2f 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1782,6 +1782,16 @@ test_oid_to_path () { echo "${1%$basename}/$basename" } +# Parse oids from git ls-files --staged output +test_parse_ls_files_stage_oids () { + awk '{print $2}' - +} + +# Parse oids from git ls-tree output +test_parse_ls_tree_oids () { + awk '{print $3}' - +} + # Choose a port number based on the test script's number and store it in # the given variable name, unless that variable already contains a number. test_set_port () { From patchwork Wed Mar 30 05:05:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75E21C433F5 for ; Wed, 30 Mar 2022 05:06:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242787AbiC3FH7 (ORCPT ); Wed, 30 Mar 2022 01:07:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242761AbiC3FHq (ORCPT ); Wed, 30 Mar 2022 01:07:46 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69F7266FBD for ; Tue, 29 Mar 2022 22:05:48 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id w4so27527429wrg.12 for ; Tue, 29 Mar 2022 22:05:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9wQouNJeYrigvBznmm6/Kim7Siohd+Q9tteLLEpJ8lg=; b=j98Uswoi6xKw7cGZXlcXGr5io8Fcz2yYUcUg8KtTPKV+HVHccb6g/gpCR0Y8mjPI/X 7qrPhCIpI69OJPc/czgjI6XknhgM8D6xZqN1TTAWAoR+neXyHuafBH5GJcqt3tJHGZOS rmF2s4ze2jdhlkM6TKvKqTQMwZejFy5xaooBoro90CKtqQKsjFyPxv4DY3MBVLwz+fjZ IUIeHht4sOx69bxPJ4OOYq91CY7BL28kJgJlb7dyLKTp5uvqL4OxOL5kJleht/4WP+NC owofBLEK30FAOPC6nYcBz2c4n6CsQortiTBoUM0i9xDIe+gOswbg2Lp5PZ+G6DB3xrtl liTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9wQouNJeYrigvBznmm6/Kim7Siohd+Q9tteLLEpJ8lg=; b=iPpbglFp0m8s6Glh5WBiP9E36Vs+Xv7nd0zTXeqUF/R/svWaPKt1nKkTtD1p5gYAD9 38dh3kcArN1a5/AeZpqkr4Q9q3CWfWp7UEBGeARzR8GxSG68zTJb351yUa70ViFtk2la DO+cryEUUjoj7eOZb1XNQ9fxE6BVuRzdJPeujwYcgd9t89WPrymX5ko67eOaQsK41Sni /UwyM7snJrgAHY0q/LTqeL/JT209I0dpLG26ZHd1lVoMaP4JDWENW7uswJHEv4aZkwvZ EubHaeMr+SzeiADHa95clPowOjrAR7dZ8tnwadqiVnolWdOxDbwKnLh1Sqg9UOVFcTna 2osQ== X-Gm-Message-State: AOAM530cKmmSB4+bkHR+aU6EZUEjwjMfLaXDkoRPYeU1224bJ5ZYM9py oMqWAOltqs6+Ow6Ob/9K+OjeD3x1tCE= X-Google-Smtp-Source: ABdhPJxMoHd+jXuqsTkdOHy6D6yMBUFQ3w+Idl0NZ06c6qrp5CUmnqiYXp/4reyIxC/RjsyXKuMwVA== X-Received: by 2002:adf:fa87:0:b0:203:f28e:76c3 with SMTP id h7-20020adffa87000000b00203f28e76c3mr34766420wrr.579.1648616746610; Tue, 29 Mar 2022 22:05:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az19-20020a05600c601300b0038cadf3aa69sm5438312wmb.36.2022.03.29.22.05.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:46 -0700 (PDT) Message-Id: <1a4aff8c350b5ffe3c7760faa4accc88c83ce11c.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:29 +0000 Subject: [PATCH v5 11/14] core.fsyncmethod: tests for batch mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Add test cases to exercise batch mode for: * 'git add' * 'git stash' * 'git update-index' * 'git unpack-objects' These tests ensure that the added data winds up in the object database. In this change we introduce a new test helper lib-unique-files.sh. The goal of this library is to create a tree of files that have different oids from any other files that may have been created in the current test repo. This helps us avoid missing validation of an object being added due to it already being in the repo. Signed-off-by: Neeraj Singh --- t/lib-unique-files.sh | 34 ++++++++++++++++++++++++++++++++++ t/t3700-add.sh | 28 ++++++++++++++++++++++++++++ t/t3903-stash.sh | 20 ++++++++++++++++++++ t/t5300-pack-object.sh | 41 +++++++++++++++++++++++++++-------------- 4 files changed, 109 insertions(+), 14 deletions(-) create mode 100644 t/lib-unique-files.sh diff --git a/t/lib-unique-files.sh b/t/lib-unique-files.sh new file mode 100644 index 00000000000..34c01a65256 --- /dev/null +++ b/t/lib-unique-files.sh @@ -0,0 +1,34 @@ +# Helper to create files with unique contents + +# Create multiple files with unique contents within this test run. Takes the +# number of directories, the number of files in each directory, and the base +# directory. +# +# test_create_unique_files 2 3 my_dir -- Creates 2 directories with 3 files +# each in my_dir, all with contents +# different from previous invocations +# of this command in this run. + +test_create_unique_files () { + test "$#" -ne 3 && BUG "3 param" + + local dirs="$1" && + local files="$2" && + local basedir="$3" && + local counter=0 && + local i && + local j && + test_tick && + local basedata=$basedir$test_tick && + rm -rf "$basedir" && + for i in $(test_seq $dirs) + do + local dir=$basedir/dir$i && + mkdir -p "$dir" && + for j in $(test_seq $files) + do + counter=$((counter + 1)) && + echo "$basedata.$counter">"$dir/file$j.txt" + done + done +} diff --git a/t/t3700-add.sh b/t/t3700-add.sh index b1f90ba3250..8979c8a5f03 100755 --- a/t/t3700-add.sh +++ b/t/t3700-add.sh @@ -8,6 +8,8 @@ test_description='Test of git add, including the -- option.' TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh +. $TEST_DIRECTORY/lib-unique-files.sh + # Test the file mode "$1" of the file "$2" in the index. test_mode_in_index () { case "$(git ls-files -s "$2")" in @@ -34,6 +36,32 @@ test_expect_success \ 'Test that "git add -- -q" works' \ 'touch -- -q && git add -- -q' +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'git add: core.fsyncmethod=batch' " + test_create_unique_files 2 4 files_base_dir1 && + GIT_TEST_FSYNC=1 git $BATCH_CONFIGURATION add -- ./files_base_dir1/ && + git ls-files --stage files_base_dir1/ | + test_parse_ls_files_stage_oids >added_files_oids && + + # We created 2 subdirs with 4 files each (8 files total) above + test_line_count = 8 added_files_oids && + git cat-file --batch-check='%(objectname)' added_files_actual && + test_cmp added_files_oids added_files_actual +" + +test_expect_success 'git update-index: core.fsyncmethod=batch' " + test_create_unique_files 2 4 files_base_dir2 && + find files_base_dir2 ! -type d -print | xargs git $BATCH_CONFIGURATION update-index --add -- && + git ls-files --stage files_base_dir2 | + test_parse_ls_files_stage_oids >added_files2_oids && + + # We created 2 subdirs with 4 files each (8 files total) above + test_line_count = 8 added_files2_oids && + git cat-file --batch-check='%(objectname)' added_files2_actual && + test_cmp added_files2_oids added_files2_actual +" + test_expect_success \ 'git add: Test that executable bit is not used if core.filemode=0' \ 'git config core.filemode 0 && diff --git a/t/t3903-stash.sh b/t/t3903-stash.sh index 4abbc8fccae..20e94881964 100755 --- a/t/t3903-stash.sh +++ b/t/t3903-stash.sh @@ -9,6 +9,7 @@ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./test-lib.sh +. $TEST_DIRECTORY/lib-unique-files.sh test_expect_success 'usage on cmd and subcommand invalid option' ' test_expect_code 129 git stash --invalid-option 2>usage && @@ -1410,6 +1411,25 @@ test_expect_success 'stash handles skip-worktree entries nicely' ' git rev-parse --verify refs/stash:A.t ' + +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'stash with core.fsyncmethod=batch' " + test_create_unique_files 2 4 files_base_dir && + GIT_TEST_FSYNC=1 git $BATCH_CONFIGURATION stash push -u -- ./files_base_dir/ && + + # The files were untracked, so use the third parent, + # which contains the untracked files + git ls-tree -r stash^3 -- ./files_base_dir/ | + test_parse_ls_tree_oids >stashed_files_oids && + + # We created 2 dirs with 4 files each (8 files total) above + test_line_count = 8 stashed_files_oids && + git cat-file --batch-check='%(objectname)' stashed_files_actual && + test_cmp stashed_files_oids stashed_files_actual +" + + test_expect_success 'git stash succeeds despite directory/file change' ' test_create_repo directory_file_switch_v1 && ( diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index a11d61206ad..f8a0f309e2d 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -161,22 +161,27 @@ test_expect_success 'pack-objects with bogus arguments' ' ' check_unpack () { + local packname="$1" && + local object_list="$2" && + local git_config="$3" && test_when_finished "rm -rf git2" && - git init --bare git2 && - git -C git2 unpack-objects -n <"$1".pack && - git -C git2 unpack-objects <"$1".pack && - (cd .git && find objects -type f -print) | - while read path - do - cmp git2/$path .git/$path || { - echo $path differs. - return 1 - } - done + git $git_config init --bare git2 && + ( + git $git_config -C git2 unpack-objects -n <"$packname".pack && + git $git_config -C git2 unpack-objects <"$packname".pack && + git $git_config -C git2 cat-file --batch-check="%(objectname)" + ) <"$object_list" >current && + cmp "$object_list" current } test_expect_success 'unpack without delta' ' - check_unpack test-1-${packname_1} + check_unpack test-1-${packname_1} obj-list +' + +BATCH_CONFIGURATION='-c core.fsync=loose-object -c core.fsyncmethod=batch' + +test_expect_success 'unpack without delta (core.fsyncmethod=batch)' ' + check_unpack test-1-${packname_1} obj-list "$BATCH_CONFIGURATION" ' test_expect_success 'pack with REF_DELTA' ' @@ -185,7 +190,11 @@ test_expect_success 'pack with REF_DELTA' ' ' test_expect_success 'unpack with REF_DELTA' ' - check_unpack test-2-${packname_2} + check_unpack test-2-${packname_2} obj-list +' + +test_expect_success 'unpack with REF_DELTA (core.fsyncmethod=batch)' ' + check_unpack test-2-${packname_2} obj-list "$BATCH_CONFIGURATION" ' test_expect_success 'pack with OFS_DELTA' ' @@ -195,7 +204,11 @@ test_expect_success 'pack with OFS_DELTA' ' ' test_expect_success 'unpack with OFS_DELTA' ' - check_unpack test-3-${packname_3} + check_unpack test-3-${packname_3} obj-list +' + +test_expect_success 'unpack with OFS_DELTA (core.fsyncmethod=batch)' ' + check_unpack test-3-${packname_3} obj-list "$BATCH_CONFIGURATION" ' test_expect_success 'compare delta flavors' ' From patchwork Wed Mar 30 05:05:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A836CC433EF for ; Wed, 30 Mar 2022 05:06:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242773AbiC3FIB (ORCPT ); Wed, 30 Mar 2022 01:08:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242762AbiC3FHq (ORCPT ); Wed, 30 Mar 2022 01:07:46 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F35B674D7 for ; Tue, 29 Mar 2022 22:05:49 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id k124-20020a1ca182000000b0038c9cf6e2a6so2735170wme.0 for ; Tue, 29 Mar 2022 22:05:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=92E+j38GPKUpnVjZkT1Mcgrfcah4Fkc3Z+GCmwZdNoI=; b=fkG4knhtX/wTVe+maOO451SWvLoSdsnaX0+D+57iLj8yc3w3cHUPsrh6mwrsOHwtQt GFsREo+Snnj8+QmqsCzaxJy9fupgyBoEU7epMoWqwf8vKWnEbJQgx6Iv1Nz+VbnlUZa/ 9kpBpBus1IN0fZsqiR4WuuCIbAf0RG01tQte+cF5vgJU6Go4fveSP1tusrxnkvMoSP03 oVKn3W0ZPvY9q6H2ePeLz2eWRJLIBCsh+Yj70eJkjUZY8cmLSYf3i024DliUVkjanZBI W3eEMKJ6NgoMcjfEBfzDfbrs836IfDNsX1F6QHdpJ0pkSqMhsyceddX1DCLV6ZUAPCWg b3ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=92E+j38GPKUpnVjZkT1Mcgrfcah4Fkc3Z+GCmwZdNoI=; b=MUn0VeTxyTZQEaWfmMU38on18EFSg9TjzIx76pSJFkEhsbuwpj2H7BpyTC59jOC9Sm LxxApSb2GQr4AnYWNx/YX6RJvRmfkKeHVLCdh9jPx2hvtVXZs/aMelV1irnZuF94TY5y VEd2ifRGUc/dP9yg5tRc2pkCncx6NhhK+aou0QL5xYS/02wBmkaByqFdAhUtpVPsSVqi i3bymW6PAkgYitayIZ8Sasiei/UX47WDgLeE1Yqgg2uph0FOXwGD3ziJOxk/dXn1EAcH rox2I3OMl8olo7P9XLu8+SvRoMJuFrqbXsImSs+fhopGqApA98mZMePZKHhn+aiDhFAz 5Wxw== X-Gm-Message-State: AOAM531M3IJhT55sr23/Bd/1YhPX1JW+P0kJuQzXh8zCfDzIrT8BWOzD O05WOSv0jVWaUKAFM4genzUe2PCak24= X-Google-Smtp-Source: ABdhPJzyqA+zcVRgCwK7EXySzhpNiQACFZGFMLJo8g+kEdDaoDAk8wFZhaVjRhOICbdUDnGGjkY81g== X-Received: by 2002:a05:600c:3b24:b0:38c:dbdb:a35c with SMTP id m36-20020a05600c3b2400b0038cdbdba35cmr2581320wms.175.1648616747677; Tue, 29 Mar 2022 22:05:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y15-20020a05600015cf00b00203e324347bsm19081430wry.102.2022.03.29.22.05.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:47 -0700 (PDT) Message-Id: <47cc63e1dda6fa6417ff6bace4e9e371aaae0d7d.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:30 +0000 Subject: [PATCH v5 12/14] t/perf: add iteration setup mechanism to perf-lib Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Tests that affect the repo in stateful ways are easier to write if we can run setup steps outside of the measured portion of perf iteration. This change adds a "--setup 'setup-script'" parameter to test_perf. To make invocations easier to understand, I also moved the prerequisites to a new --prereq parameter. The setup facility will be used in the upcoming perf tests for batch mode, but it already helps in some existing tests, like t5302 and t7820. Signed-off-by: Neeraj Singh --- t/perf/p4220-log-grep-engines.sh | 3 +- t/perf/p4221-log-grep-engines-fixed.sh | 3 +- t/perf/p5302-pack-index.sh | 15 +++---- t/perf/p7519-fsmonitor.sh | 18 ++------ t/perf/p7820-grep-engines.sh | 6 ++- t/perf/perf-lib.sh | 62 +++++++++++++++++++++++--- 6 files changed, 74 insertions(+), 33 deletions(-) diff --git a/t/perf/p4220-log-grep-engines.sh b/t/perf/p4220-log-grep-engines.sh index 2bc47ded4d1..03fbfbb85d3 100755 --- a/t/perf/p4220-log-grep-engines.sh +++ b/t/perf/p4220-log-grep-engines.sh @@ -36,7 +36,8 @@ do else prereq="" fi - test_perf $prereq "$engine log$GIT_PERF_4220_LOG_OPTS --grep='$pattern'" " + test_perf "$engine log$GIT_PERF_4220_LOG_OPTS --grep='$pattern'" \ + --prereq "$prereq" " git -c grep.patternType=$engine log --pretty=format:%h$GIT_PERF_4220_LOG_OPTS --grep='$pattern' >'out.$engine' || : " done diff --git a/t/perf/p4221-log-grep-engines-fixed.sh b/t/perf/p4221-log-grep-engines-fixed.sh index 060971265a9..0a6d6dfc219 100755 --- a/t/perf/p4221-log-grep-engines-fixed.sh +++ b/t/perf/p4221-log-grep-engines-fixed.sh @@ -26,7 +26,8 @@ do else prereq="" fi - test_perf $prereq "$engine log$GIT_PERF_4221_LOG_OPTS --grep='$pattern'" " + test_perf "$engine log$GIT_PERF_4221_LOG_OPTS --grep='$pattern'" \ + --prereq "$prereq" " git -c grep.patternType=$engine log --pretty=format:%h$GIT_PERF_4221_LOG_OPTS --grep='$pattern' >'out.$engine' || : " done diff --git a/t/perf/p5302-pack-index.sh b/t/perf/p5302-pack-index.sh index c16f6a3ff69..14c601bbf86 100755 --- a/t/perf/p5302-pack-index.sh +++ b/t/perf/p5302-pack-index.sh @@ -26,9 +26,8 @@ test_expect_success 'set up thread-counting tests' ' done ' -test_perf PERF_EXTRA 'index-pack 0 threads' ' - rm -rf repo.git && - git init --bare repo.git && +test_perf 'index-pack 0 threads' --prereq PERF_EXTRA \ + --setup 'rm -rf repo.git && git init --bare repo.git' ' GIT_DIR=repo.git git index-pack --threads=1 --stdin < $PACK ' @@ -36,17 +35,15 @@ for t in $threads do THREADS=$t export THREADS - test_perf PERF_EXTRA "index-pack $t threads" ' - rm -rf repo.git && - git init --bare repo.git && + test_perf "index-pack $t threads" --prereq PERF_EXTRA \ + --setup 'rm -rf repo.git && git init --bare repo.git' ' GIT_DIR=repo.git GIT_FORCE_THREADS=1 \ git index-pack --threads=$THREADS --stdin <$PACK ' done -test_perf 'index-pack default number of threads' ' - rm -rf repo.git && - git init --bare repo.git && +test_perf 'index-pack default number of threads' \ + --setup 'rm -rf repo.git && git init --bare repo.git' ' GIT_DIR=repo.git git index-pack --stdin < $PACK ' diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index c8be58f3c76..5b489c968b8 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -60,18 +60,6 @@ then esac fi -if test -n "$GIT_PERF_7519_DROP_CACHE" -then - # When using GIT_PERF_7519_DROP_CACHE, GIT_PERF_REPEAT_COUNT must be 1 to - # generate valid results. Otherwise the caching that happens for the nth - # run will negate the validity of the comparisons. - if test "$GIT_PERF_REPEAT_COUNT" -ne 1 - then - echo "warning: Setting GIT_PERF_REPEAT_COUNT=1" >&2 - GIT_PERF_REPEAT_COUNT=1 - fi -fi - trace_start() { if test -n "$GIT_PERF_7519_TRACE" then @@ -167,10 +155,10 @@ setup_for_fsmonitor() { test_perf_w_drop_caches () { if test -n "$GIT_PERF_7519_DROP_CACHE"; then - test-tool drop-caches + test_perf "$1" --setup "test-tool drop-caches" "$2" + else + test_perf "$@" fi - - test_perf "$@" } test_fsmonitor_suite() { diff --git a/t/perf/p7820-grep-engines.sh b/t/perf/p7820-grep-engines.sh index 8b09c5bf328..9bfb86842a9 100755 --- a/t/perf/p7820-grep-engines.sh +++ b/t/perf/p7820-grep-engines.sh @@ -49,13 +49,15 @@ do fi if ! test_have_prereq PERF_GREP_ENGINES_THREADS then - test_perf $prereq "$engine grep$GIT_PERF_7820_GREP_OPTS '$pattern'" " + test_perf "$engine grep$GIT_PERF_7820_GREP_OPTS '$pattern'" \ + --prereq "$prereq" " git -c grep.patternType=$engine grep$GIT_PERF_7820_GREP_OPTS -- '$pattern' >'out.$engine' || : " else for threads in $GIT_PERF_GREP_THREADS do - test_perf PTHREADS,$prereq "$engine grep$GIT_PERF_7820_GREP_OPTS '$pattern' with $threads threads" " + test_perf "$engine grep$GIT_PERF_7820_GREP_OPTS '$pattern' with $threads threads" + --prereq PTHREADS,$prereq " git -c grep.patternType=$engine -c grep.threads=$threads grep$GIT_PERF_7820_GREP_OPTS -- '$pattern' >'out.$engine.$threads' || : " done diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 407252bac70..e9cc16d045a 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -189,19 +189,40 @@ exit $ret' >&3 2>&4 } test_wrapper_ () { + local test_wrapper_func_ test_title_ test_wrapper_func_=$1; shift + test_title_=$1; shift test_start_ - test "$#" = 3 && { test_prereq=$1; shift; } || test_prereq= - test "$#" = 2 || - BUG "not 2 or 3 parameters to test-expect-success" + test_prereq= + test_perf_setup_= + while test $# != 0 + do + case $1 in + --prereq) + test_prereq=$2 + shift + ;; + --setup) + test_perf_setup_=$2 + shift + ;; + *) + break + ;; + esac + shift + done + test "$#" = 1 || BUG "test_wrapper_ needs 2 positional parameters" export test_prereq - if ! test_skip "$@" + export test_perf_setup_ + + if ! test_skip "$test_title_" "$@" then base=$(basename "$0" .sh) echo "$test_count" >>"$perf_results_dir"/$base.subtests echo "$1" >"$perf_results_dir"/$base.$test_count.descr base="$perf_results_dir"/"$PERF_RESULTS_PREFIX$(basename "$0" .sh)"."$test_count" - "$test_wrapper_func_" "$@" + "$test_wrapper_func_" "$test_title_" "$@" fi test_finish_ @@ -214,6 +235,16 @@ test_perf_ () { echo "perf $test_count - $1:" fi for i in $(test_seq 1 $GIT_PERF_REPEAT_COUNT); do + if test -n "$test_perf_setup_" + then + say >&3 "setup: $test_perf_setup_" + if ! test_eval_ $test_perf_setup_ + then + test_failure_ "$test_perf_setup_" + break + fi + + fi say >&3 "running: $2" if test_run_perf_ "$2" then @@ -237,11 +268,24 @@ test_perf_ () { rm test_time.* } +# Usage: test_perf 'title' [options] 'perf-test' +# Run the performance test script specified in perf-test with +# optional prerequisite and setup steps. +# Options: +# --prereq prerequisites: Skip the test if prequisites aren't met +# --setup "setup-steps": Run setup steps prior to each measured iteration +# test_perf () { test_wrapper_ test_perf_ "$@" } test_size_ () { + if test -n "$test_perf_setup_" + then + say >&3 "setup: $test_perf_setup_" + test_eval_ $test_perf_setup_ + fi + say >&3 "running: $2" if test_eval_ "$2" 3>"$base".result; then test_ok_ "$1" @@ -250,6 +294,14 @@ test_size_ () { fi } +# Usage: test_size 'title' [options] 'size-test' +# Run the size test script specified in size-test with optional +# prerequisites and setup steps. Returns the numeric value +# returned by size-test. +# Options: +# --prereq prerequisites: Skip the test if prequisites aren't met +# --setup "setup-steps": Run setup steps prior to the size measurement + test_size () { test_wrapper_ test_size_ "$@" } From patchwork Wed Mar 30 05:05:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12795458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19EAEC433EF for ; Wed, 30 Mar 2022 05:06:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242823AbiC3FIG (ORCPT ); Wed, 30 Mar 2022 01:08:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242767AbiC3FHr (ORCPT ); Wed, 30 Mar 2022 01:07:47 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C08866CA9 for ; Tue, 29 Mar 2022 22:05:50 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id bg31-20020a05600c3c9f00b00381590dbb33so2704275wmb.3 for ; Tue, 29 Mar 2022 22:05:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Ws+waIF2WcOuQF4QexmvEEe26McYMmSbQKCcdTaB9zw=; b=pig1jJoN3XpM+REuvdNkUz0gWfRMV+hbtilTeKsN7uMFLj2s6UdTM111iT+EcAqXoz VnbznXxkjR0xRXt/P9kNcWO3j03Ex09kx4Jvxfz7tMRGa/UinxZYnSYlIZvMz7CqEdHq aEeVYUhXXtS1ELBaxD0oVvtQw4JJsNAMVFTO4o7yzzUWMs98j8X+jGLhXKAx7JTqT8RD 9158Rit1oxufxHZtC40x4Bzlpbr7sZPXqTKYVLWRdQAbOXJ+HIrwIBCLi8QsIwIW3lfY atvmMqQa/hBg9jJNPlVzxEXveVd+MaZCRvFt3Df/YweMw1dADLyPK/xRbiwx4PmfCQME ee1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Ws+waIF2WcOuQF4QexmvEEe26McYMmSbQKCcdTaB9zw=; b=htG+1q8OMzAYgPHVmhPydKFbgz3hgYq6EQK1xCy8tQbghTAWHk1t2vzJsE8VIuyZ5N aF+SWVlSpSVnJJoZQbsruSBgUNfeHq5cV2TIup+2Y32F+hrPcx7zrJPvy5siJY0q/+oG cEb3d6Dr2yOhAQ9mmCZjHJEtmbzFEnloEsvlrQ0BDrQQMYRWt72CsG9NXxuMjFnl7VL/ WybbbE5G/YulAK9iLDcjHwgmBNPEk7acrPcVObiXIuXBNGtQqnuLSd326VWRBWhv5TwE 61bqf9ejjqCJXFQz389u1Z8nRvPtCiNJqM+d8SLqGkNa54/9Qgmdun0E4J/1DfXlsWqn WDDw== X-Gm-Message-State: AOAM533NmluY8UTEvAsHgT5/gXcfMurDkCAOBTyIHGM/H5dXXGunbVEb NyOGWdyzmGPpV/m/S++CJQw5CWcVxYQ= X-Google-Smtp-Source: ABdhPJxdLjVWDpzowHsSd0DL2OM4+Jzo7MFUFIpmCjjXoWNuQVnmpwK3R0HUAQ8n9LAeXsN/x41sEw== X-Received: by 2002:a05:600c:207:b0:38c:c815:8043 with SMTP id 7-20020a05600c020700b0038cc8158043mr2674655wmi.35.1648616748648; Tue, 29 Mar 2022 22:05:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j8-20020a05600c404800b0038cc9c7670bsm3685715wmm.3.2022.03.29.22.05.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:48 -0700 (PDT) Message-Id: <26be6ecb28bc1f76fba380fdd10acf59820df997.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:31 +0000 Subject: [PATCH v5 13/14] core.fsyncmethod: performance tests for batch mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Add basic performance tests for git commands that can add data to the object database. We cover: * git add * git stash * git update-index (via git stash) * git unpack-objects * git commit --all We cover all currently available fsync methods as well. Signed-off-by: Neeraj Singh --- t/perf/p0008-odb-fsync.sh | 81 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100755 t/perf/p0008-odb-fsync.sh diff --git a/t/perf/p0008-odb-fsync.sh b/t/perf/p0008-odb-fsync.sh new file mode 100755 index 00000000000..87092c2627e --- /dev/null +++ b/t/perf/p0008-odb-fsync.sh @@ -0,0 +1,81 @@ +#!/bin/sh +# +# This test measures the performance of adding new files to the object +# database. The test was originally added to measure the effect of the +# core.fsyncMethod=batch mode, which is why we are testing different values of +# that setting explicitly and creating a lot of unique objects. + +test_description="Tests performance of adding things to the object database" + +. ./perf-lib.sh + +. $TEST_DIRECTORY/lib-unique-files.sh + +test_perf_fresh_repo +test_checkout_worktree + +dir_count=10 +files_per_dir=50 +total_files=$((dir_count * files_per_dir)) + +populate_files () { + test_create_unique_files $dir_count $files_per_dir files +} + +setup_repo () { + (rm -rf .git || 1) && + git init && + test_commit first && + populate_files +} + +test_perf_fsync_cfgs () { + local method cfg && + for method in none fsync batch writeout-only + do + case $method in + none) + cfg="-c core.fsync=none" + ;; + *) + cfg="-c core.fsync=loose-object -c core.fsyncMethod=$method" + esac && + + # Set GIT_TEST_FSYNC=1 explicitly since fsync is normally + # disabled by t/test-lib.sh. + if ! test_perf "$1 (fsyncMethod=$method)" \ + --setup "$2" \ + "GIT_TEST_FSYNC=1 git $cfg $3" + then + break + fi + done +} + +test_perf_fsync_cfgs "add $total_files files" \ + "setup_repo" \ + "add -- files" + +test_perf_fsync_cfgs "stash $total_files files" \ + "setup_repo" \ + "stash push -u -- files" + +test_perf_fsync_cfgs "unpack $total_files files" \ + " + setup_repo && + git -c core.fsync=none add -- files && + git -c core.fsync=none commit -q -m second && + echo HEAD | git pack-objects -q --stdout --revs >test_pack.pack && + setup_repo + " \ + "unpack-objects -q X-Patchwork-Id: 12795459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 789ABC433F5 for ; Wed, 30 Mar 2022 05:06:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242825AbiC3FIJ (ORCPT ); Wed, 30 Mar 2022 01:08:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242775AbiC3FHr (ORCPT ); Wed, 30 Mar 2022 01:07:47 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2DC768332 for ; Tue, 29 Mar 2022 22:05:51 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id j18so27543959wrd.6 for ; Tue, 29 Mar 2022 22:05:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/dJLnMNPf593DDA+ox52qw4qYfm4mKhrA2SM6fu1eys=; b=I9YpkH5faX8/V+akBN1uC2UV72NHOvnYSftgPDgUvHe4Sfh0+Z88nW0svaz43DpZ31 OTONCfp5O55I2ckAL6GaZn4IOc0O8VaZ8/OoO0Q7pU9Ns/NeH13EzHHl60hPy5+FEhz5 LMoEjs+0vmCSPfYDEEbD6Lb2VJENggFPunlQTEXVML7KANHwd3e3xEFvgIzgCE8AfVtg 3GcsobXuoW2Y/zlNdVlaY7jepkb8PCGVTcags9mvc6tzRbroOp2SKZ+i/2RrXKm38tYU Qs8TnMHeDP6h45v1lqAOxzPeq1PbbLtsvzQzF/m+g/PaYMV9UnF+pZd+S2PE9OYjr2lU Y7pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/dJLnMNPf593DDA+ox52qw4qYfm4mKhrA2SM6fu1eys=; b=B/Rcd9CmUYFXCUub927e32zgZoGZJKde6vlMfdTMqjqd1E9RYMgYp6k1E6z/PaJbrg 7ipPlDl5dqVFkW/zfBwHqU64NiQNZFYKPNVYArd8kElMTuYLFi17IikpK1vgBN5hiUS0 Ue6n3+iLge37fs9c5AArVh2jX9bKLwvpvuMz7ucFCPvetWrH0DnBKb0SZX9odIJW6vdr zVgxXogohrc8Uxqbh34lm86E3XCXxwBpSX4li3bUqYQFGp1OmlJ0rUMbXsPNtYnhbD8o mSnMMzuEFInG9mzrFhzHvlnqwac4RMVL2IAsVYjhG5/I0OwEoJFT1ANWOHPFBiYEnBuL H0fQ== X-Gm-Message-State: AOAM531VJfeZTR1OLeOTJi4RvgS3fDP4yQ8plnv63qdG2jlkwuuX+G7O MB8V6tnMLDtNkTz2KpnsIzBjlsbu+Gc= X-Google-Smtp-Source: ABdhPJzbNfjZMA8s65SOAfcx+AXxoUE1ZgYdvy2XWUH/2XYSG62HKRQoRceFbjXk34v4UikfDGEa7Q== X-Received: by 2002:a5d:6208:0:b0:203:dde4:c76e with SMTP id y8-20020a5d6208000000b00203dde4c76emr34897214wru.273.1648616749538; Tue, 29 Mar 2022 22:05:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k11-20020a5d6d4b000000b0020599079f68sm15357101wri.106.2022.03.29.22.05.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Mar 2022 22:05:49 -0700 (PDT) Message-Id: <88c1f84d4c3f71bb3cbd6e016771196a56f90bd9.1648616734.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 30 Mar 2022 05:05:32 +0000 Subject: [PATCH v5 14/14] core.fsyncmethod: correctly camel-case warning message Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Johannes.Schindelin@gmx.de, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, jeffhost@microsoft.com, Bagas Sanjaya , worldhello.net@gmail.com, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The warning for an unrecognized fsyncMethod was not camel-cased. Signed-off-by: Neeraj Singh --- config.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config.c b/config.c index e9cac5f4707..ae819dee20b 100644 --- a/config.c +++ b/config.c @@ -1697,7 +1697,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) if (!strcmp(var, "core.fsyncobjectfiles")) { if (fsync_object_files < 0) - warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); + warning(_("core.fsyncObjectFiles is deprecated; use core.fsync instead")); fsync_object_files = git_config_bool(var, value); return 0; }