From patchwork Fri Aug 27 23:49:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12462965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0D9EC4320E for ; Fri, 27 Aug 2021 23:49:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C780160EBA for ; Fri, 27 Aug 2021 23:49:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232600AbhH0Xub (ORCPT ); Fri, 27 Aug 2021 19:50:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232547AbhH0Xua (ORCPT ); Fri, 27 Aug 2021 19:50:30 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D058C061796 for ; Fri, 27 Aug 2021 16:49:40 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id y195so1974861wmc.2 for ; Fri, 27 Aug 2021 16:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vnlYrcMPlZV7jn+Lh11sIHW9c4SJp4J6AKtj4Z4AN0M=; b=hPSy3PTiyJIQx+RD7TYCyRwCXCvwYExuJ1YG/9d22YlnJelBUd51kwJh4f/HXkqlYn j2ZUchkCr01Q4JEXh6pdjmCjXrCyCAfp9Ucne0oTdjwyCw2OcJG/bXNLdJ+iWPBCpprh eXQd/meoPeMOEk2LJ5Ey/P7g38HFAMSEfel/MSzCbkBbdJw2Oo3B0HclcuGVzAKsr+3f trovRZAKv+BqMm7A7xCfLL7uDi4aic/YeZxEBDW8GIdX8J2JBb+yJlUCOtx20uJ8/RmE swaK7mvJsb8cGXVSWepJOEyl9MBjqK9W3c4bhAx5zY13QpYM6RFhC/jTl5Nmvho/9W1P GP1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vnlYrcMPlZV7jn+Lh11sIHW9c4SJp4J6AKtj4Z4AN0M=; b=l6r3SK/i7I+M0akyBnQJ7ZO++hCVAXb6Wi7bCh5g22743VpcEOE8OHVlgRQzKHnuAo HkoSFbIrqJeu+UGJOeDLIkJfhrWy/x2eiM7UePvPLg9j6FVF28FwI6dJGptW7JHkwE0L l2D5wmykX/++FgsF0VPx8OrFjVVYk1H0SNCw8okrW6RzklkCCO66pTBUmSNWGi12GYxl GyGBf79VNoOn0A0ynWX4XFxWgMTOvcjxg0AJf5A5hFPnv2bG95B8F77+QNtkajDk7X+d 0s07if4PR/vIfyBYHK6XmZtgBPihQpNs5EUqKsQoszMxJp2xpRWmbGlLximIrlR06vhR MRGA== X-Gm-Message-State: AOAM530yf8Kx4FGMKEzkYFo4cHJHzq2tWJxifAjrIfxiD635Ov/jfycf oFzBw5YmQ3fEgbvJTfWNcz8bDxeGsZc= X-Google-Smtp-Source: ABdhPJxzwRd3M0pfxtvirRAbKYxBKhQLnYp1E1On/QKhg4/NCIFVsop10LVOOaO1ZE/3fkBMRbV9tQ== X-Received: by 2002:a1c:f315:: with SMTP id q21mr11130015wmq.76.1630108178952; Fri, 27 Aug 2021 16:49:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k1sm8022099wrz.61.2021.08.27.16.49.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Aug 2021 16:49:38 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 27 Aug 2021 23:49:32 +0000 Subject: [PATCH v2 1/6] object-file: use futimens rather than utime Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Neeraj-Personal , Johannes Schindelin , Jeff King , Jeff Hostetler , Christoph Hellwig , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Make close_loose_object do all of the steps for syncing and correctly naming a new loose object so that it can be reimplemented in the upcoming bulk-fsync mode. Use futimens, which is available in POSIX.1-2008 to update the file timestamps. This should be slightly faster than utime, since we have a file descriptor already available. This change allows us to update the time before closing, renaming, and potentially fsyincing the file being refreshed. This code is currently only invoked by git-pack-objects via force_object_loose. Implement a futimens shim for the Windows port of Git. Signed-off-by: Neeraj Singh --- compat/mingw.c | 53 ++++++++++++++++++++++++++++++++++---------------- compat/mingw.h | 2 ++ object-file.c | 17 ++++++++-------- 3 files changed, 46 insertions(+), 26 deletions(-) diff --git a/compat/mingw.c b/compat/mingw.c index 9e0cd1e097f..ce14b21c182 100644 --- a/compat/mingw.c +++ b/compat/mingw.c @@ -734,11 +734,14 @@ int mingw_chmod(const char *filename, int mode) * The unit of FILETIME is 100-nanoseconds since January 1, 1601, UTC. * Returns the 100-nanoseconds ("hekto nanoseconds") since the epoch. */ + +#define UNIX_EPOCH_FILETIME 116444736000000000LL + static inline long long filetime_to_hnsec(const FILETIME *ft) { long long winTime = ((long long)ft->dwHighDateTime << 32) + ft->dwLowDateTime; /* Windows to Unix Epoch conversion */ - return winTime - 116444736000000000LL; + return winTime - UNIX_EPOCH_FILETIME; } static inline void filetime_to_timespec(const FILETIME *ft, struct timespec *ts) @@ -748,6 +751,13 @@ static inline void filetime_to_timespec(const FILETIME *ft, struct timespec *ts) ts->tv_nsec = (hnsec % 10000000) * 100; } +static inline void timespec_to_filetime(const struct timespec *t, FILETIME *ft) +{ + long long winTime = t->tv_sec * 10000000LL + t->tv_nsec / 100 + UNIX_EPOCH_FILETIME; + ft->dwLowDateTime = winTime; + ft->dwHighDateTime = winTime >> 32; +} + /** * Verifies that safe_create_leading_directories() would succeed. */ @@ -949,19 +959,33 @@ int mingw_fstat(int fd, struct stat *buf) } } -static inline void time_t_to_filetime(time_t t, FILETIME *ft) +int mingw_futimens(int fd, const struct timespec times[2]) { - long long winTime = t * 10000000LL + 116444736000000000LL; - ft->dwLowDateTime = winTime; - ft->dwHighDateTime = winTime >> 32; + FILETIME mft, aft; + + if (times) { + timespec_to_filetime(×[0], &aft); + timespec_to_filetime(×[1], &mft); + } else { + GetSystemTimeAsFileTime(&mft); + aft = mft; + } + + if (!SetFileTime((HANDLE)_get_osfhandle(fd), NULL, &aft, &mft)) { + errno = EINVAL; + return -1; + } + + return 0; } -int mingw_utime (const char *file_name, const struct utimbuf *times) +int mingw_utime(const char *file_name, const struct utimbuf *times) { - FILETIME mft, aft; int fh, rc; DWORD attrs; wchar_t wfilename[MAX_PATH]; + struct timespec ts[2]; + if (xutftowcs_path(wfilename, file_name) < 0) return -1; @@ -979,17 +1003,12 @@ int mingw_utime (const char *file_name, const struct utimbuf *times) } if (times) { - time_t_to_filetime(times->modtime, &mft); - time_t_to_filetime(times->actime, &aft); - } else { - GetSystemTimeAsFileTime(&mft); - aft = mft; + memset(ts, 0, sizeof(ts)); + ts[0].tv_sec = times->actime; + ts[1].tv_sec = times->modtime; } - if (!SetFileTime((HANDLE)_get_osfhandle(fh), NULL, &aft, &mft)) { - errno = EINVAL; - rc = -1; - } else - rc = 0; + + rc = mingw_futimens(fh, times ? ts : NULL); close(fh); revert_attrs: diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..87944dfec72 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -398,6 +398,8 @@ int mingw_fstat(int fd, struct stat *buf); int mingw_utime(const char *file_name, const struct utimbuf *times); #define utime mingw_utime +int mingw_futimens(int fd, const struct timespec times[2]); +#define futimens mingw_futimens size_t mingw_strftime(char *s, size_t max, const char *format, const struct tm *tm); #define strftime mingw_strftime diff --git a/object-file.c b/object-file.c index a8be8994814..5421811273e 100644 --- a/object-file.c +++ b/object-file.c @@ -1860,12 +1860,13 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, } /* Finalize a file on disk, and close it. */ -static void close_loose_object(int fd) +static int close_loose_object(int fd, const char *tmpfile, const char *filename) { if (fsync_object_files) fsync_or_die(fd, "loose object file"); if (close(fd) != 0) die_errno(_("error when closing loose object file")); + return finalize_object_file(tmpfile, filename); } /* Size of directory component, including the ending '/' */ @@ -1973,17 +1974,15 @@ static int write_loose_object(const struct object_id *oid, char *hdr, die(_("confused by unstable object source data for %s"), oid_to_hex(oid)); - close_loose_object(fd); - if (mtime) { - struct utimbuf utb; - utb.actime = mtime; - utb.modtime = mtime; - if (utime(tmp_file.buf, &utb) < 0) - warning_errno(_("failed utime() on %s"), tmp_file.buf); + struct timespec ts[2] = {0}; + ts[0].tv_sec = mtime; + ts[1].tv_sec = mtime; + if (futimens(fd, ts) < 0) + warning_errno(_("failed futimes() on %s"), tmp_file.buf); } - return finalize_object_file(tmp_file.buf, filename.buf); + return close_loose_object(fd, tmp_file.buf, filename.buf); } static int freshen_loose_object(const struct object_id *oid) From patchwork Fri Aug 27 23:49:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12462967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46FA6C4320A for ; Fri, 27 Aug 2021 23:49:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B94B60F91 for ; Fri, 27 Aug 2021 23:49:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232628AbhH0Xue (ORCPT ); Fri, 27 Aug 2021 19:50:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232546AbhH0Xua (ORCPT ); Fri, 27 Aug 2021 19:50:30 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF9FCC0613D9 for ; Fri, 27 Aug 2021 16:49:40 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id g18so5236225wrc.11 for ; Fri, 27 Aug 2021 16:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7X2vP95Nrih2/ZYF5MeQkFO3N7Ybz8M6MMt6amHXYUA=; b=ROIPV3pSetfdE0UuaKDjco2qLyaOmwySA8tk2oHLXkk6kKZmSqpbTYedq9oTbCodHB HlD50AcMcNUcu7QDx9byfwZKY6vDlf4tSsolq6xDM3RSWXhbZGi3+uP0Tt55JZju3B8M ZmtI8kGvAeAi3fct2aP71eyB6Ioii/wuRT6jrCN8vlJEHLKerturaBx8QYXLhuXoz7wZ KLDKAv8sXEXn6p8SpOBhcBBFJL0aIbBb+xMOi1mjz3HN288ugGB44rk/2P6ZW8iuPH7t PPvZEmyQ/AYqxtvk3P8LamxGpfrt9lTdBC0zzFYcxKwCXz/nwhhLxl3fVUkb/UfWAa6s w/5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7X2vP95Nrih2/ZYF5MeQkFO3N7Ybz8M6MMt6amHXYUA=; b=HV4CCVxp0/01gGqPfo1mG0vBzNWKp9N5fDyeKuxzxDt2KxyVqMAimuHyDMcyfBnWDW V7UnrdEsI3BVI+BEL0Iy+MhUgUvU36CS/GjK8f4GVzygAaty1DibecxvBdD5t+8sBRgX RBLA4Ff4OhzFPlPfynYZ9mKZ7q2CCcZE/w0iD71zBp5emBqHDImPZ7IdFjG+C8N/EPA4 HfpT1Ax0PTB8EgK21FVudvXK8mij8B8mabexZbqsR0DF//JadhazhBtvbCZ4InEuu+B7 4H+Hcuzbwu5GA2PSJ4nyTodXDWzTwxEIA54o9vIpHX4MXjur+tNDdWQqplwt1XeoMRvR lcWw== X-Gm-Message-State: AOAM5321XAdA2Z4RdehsvFC/XcWVO8ScajEFFYoXdtYIX8mXKmxJPM4B zyosMNydr4fKZWO83uNG/Do+6OJ6DP8= X-Google-Smtp-Source: ABdhPJxYTI7g5g7SDJrwJzVtstIlV2kXZaJsmy+k9NJqlU4vWMIVlu+xL+pTC2IHj2+NnQ3Sb0XMZg== X-Received: by 2002:adf:c148:: with SMTP id w8mr5594934wre.317.1630108179523; Fri, 27 Aug 2021 16:49:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c13sm7697581wru.73.2021.08.27.16.49.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Aug 2021 16:49:39 -0700 (PDT) Message-Id: <49f72800bfb9d23d5f42005f494a56bbabdf4aca.1630108177.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 27 Aug 2021 23:49:33 +0000 Subject: [PATCH v2 2/6] bulk-checkin: rename 'state' variable and separate 'plugged' boolean Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Neeraj-Personal , Johannes Schindelin , Jeff King , Jeff Hostetler , Christoph Hellwig , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Preparation for adding bulk-fsync to the bulk-checkin.c infrastructure. * Rename 'state' variable to 'bulk_checkin_state', since we will later be adding 'bulk_fsync_state'. This also makes the variable easier to find in the debugger, since the name is more unique. * Move the 'plugged' data member of 'bulk_checkin_state' into a separate static variable. Doing this avoids resetting the variable in finish_bulk_checkin when zeroing the 'bulk_checkin_state'. As-is, we seem to unintentionally disable the plugging functionality the first time a new packfile must be created due to packfile size limits. While disabling the plugging state only results in suboptimal behavior for the current code, it would be fatal for the bulk-fsync functionality later in this patch series. Signed-off-by: Neeraj Singh --- bulk-checkin.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/bulk-checkin.c b/bulk-checkin.c index b023d9959aa..f117d62c908 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -10,9 +10,9 @@ #include "packfile.h" #include "object-store.h" -static struct bulk_checkin_state { - unsigned plugged:1; +static int bulk_checkin_plugged; +static struct bulk_checkin_state { char *pack_tmp_name; struct hashfile *f; off_t offset; @@ -21,7 +21,7 @@ static struct bulk_checkin_state { struct pack_idx_entry **written; uint32_t alloc_written; uint32_t nr_written; -} state; +} bulk_checkin_state; static void finish_bulk_checkin(struct bulk_checkin_state *state) { @@ -260,21 +260,23 @@ int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags) { - int status = deflate_to_pack(&state, oid, fd, size, type, + int status = deflate_to_pack(&bulk_checkin_state, oid, fd, size, type, path, flags); - if (!state.plugged) - finish_bulk_checkin(&state); + if (!bulk_checkin_plugged) + finish_bulk_checkin(&bulk_checkin_state); return status; } void plug_bulk_checkin(void) { - state.plugged = 1; + assert(!bulk_checkin_plugged); + bulk_checkin_plugged = 1; } void unplug_bulk_checkin(void) { - state.plugged = 0; - if (state.f) - finish_bulk_checkin(&state); + assert(bulk_checkin_plugged); + bulk_checkin_plugged = 0; + if (bulk_checkin_state.f) + finish_bulk_checkin(&bulk_checkin_state); } From patchwork Fri Aug 27 23:49:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12462975 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64EDEC4320E for ; Fri, 27 Aug 2021 23:49:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 40B9960EBA for ; Fri, 27 Aug 2021 23:49:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232627AbhH0Xuf (ORCPT ); Fri, 27 Aug 2021 19:50:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232589AbhH0Xub (ORCPT ); Fri, 27 Aug 2021 19:50:31 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0D56C0613D9 for ; Fri, 27 Aug 2021 16:49:41 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id v20-20020a1cf714000000b002e71f4d2026so8133243wmh.1 for ; Fri, 27 Aug 2021 16:49:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MwyAq5tQxJQ7clw9aLkPfVMTXXOJE6Nf8ZujzYuNAfk=; b=QQ4WwXJOlVnDvT/M0SdI/ZfTGj9MXnH8eENfQxebJt+ZtudMhGW+agxO5LSMnawF9t YGHfIgHhjoZiudDbtF+DwxLi9aEpwzyeZbO1o6qgnAT69//24x4c3ZmkW5Ckii3+pTyS nWKx1PYeS2K32x3Pz6EV2gVf+OdCDrhyeKwgVODnxdO24uwBe0wjcfBxUs9SKUNpEur6 m55lA2okHkATLZcigoi/YqVlZ4vrquhJZhGWxBvJFNHsImS+jR+FqOle+xGXQJUucm9/ ndVCBWQP4r352HbS9DPA9QNz2TTuT1mC2Nun/IIgmSTSuficRp1V9zVgoUdEymX/AYsm WHSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MwyAq5tQxJQ7clw9aLkPfVMTXXOJE6Nf8ZujzYuNAfk=; b=hSsvcfdwXroKSp+p7l/YCMl65whZUQoBltaL/nRAe4kmEs74HxDihIzri0u2C4UW6O Gm8NwioOpCX5p+QIWy2e4mnKoXbg2Ekh7mRsi3kyNi7Mm4Jhqo0HH+StodZ43vkkiHvg TgNU59lx/r1XaHIOxiC+im6C8uCvmvaj6yiewFyMnYSZkoEKCxBQvz8Rv6IEaPr9BTYd /ys5bH3Q1+dT24ipUfID4sCsauNZ9SOA7gt36zJc9y0a2ULZEIkrMvci6ZLcNG+BAJnG 3BnVkgYXXzPRIQztDLg7QC4xGEz17N1Y8rSgZLk6QdDHeBZJ8iqLnX7DTlAWHevaOKcy 7RdQ== X-Gm-Message-State: AOAM5304oQ5uUsdLgZ3u5YuS05lDLyPBZ2cnxLZ4jznkVLLXuxTRD//k ZoV4a533BqO3v6db29DiBMb7xTRDeHU= X-Google-Smtp-Source: ABdhPJxXvzued8LUl4T4TEONtREZvrgJTUvFIN/8956qb7uQ7s69azkAY169F5XJ0EDZOoVCpLagCg== X-Received: by 2002:a05:600c:3397:: with SMTP id o23mr20701714wmp.38.1630108180092; Fri, 27 Aug 2021 16:49:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g5sm7705277wrq.80.2021.08.27.16.49.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Aug 2021 16:49:39 -0700 (PDT) Message-Id: <2c1c907b12a5a8592136dcf775c9c61fac986952.1630108177.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 27 Aug 2021 23:49:34 +0000 Subject: [PATCH v2 3/6] core.fsyncobjectfiles: batched disk flushes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Neeraj-Personal , Johannes Schindelin , Jeff King , Jeff Hostetler , Christoph Hellwig , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh When adding many objects to a repo with core.fsyncObjectFiles set to true, the cost of fsync'ing each object file can become prohibitive. One major source of the cost of fsync is the implied flush of the hardware writeback cache within the disk drive. Fortunately, Windows, macOS, and Linux each offer mechanisms to write data from the filesystem page cache without initiating a hardware flush. This patch introduces a new 'core.fsyncObjectFiles = batch' option that takes advantage of the bulk-checkin infrastructure to batch up hardware flushes. When the new mode is enabled we do the following for new objects: 1. Create a tmp_obj_XXXX file and write the object data to it. 2. Issue a pagecache writeback request and wait for it to complete. 3. Record the tmp name and the final name in the bulk-checkin state for later rename. At the end of the entire transaction we: 1. Issue a fsync against the lock file to flush the hardware writeback cache, which should by now have processed the tmp file writes. 2. Rename all of the temp files to their final names. 3. When updating the index and/or refs, we assume that Git will issue another fsync internal to that operation. On a filesystem with a singular journal that is updated during name operations (e.g. create, link, rename, etc), such as NTFS and HFS+, we would expect the fsync to trigger a journal writeout so that this sequence is enough to ensure that the user's data is durable by the time the git command returns. This change also updates the macOS code to trigger a real hardware flush via fnctl(fd, F_FULLFSYNC) when fsync_or_die is called. Previously, on macOS there was no guarantee of durability since a simple fsync(2) call does not flush any hardware caches. _Performance numbers_: Linux - Hyper-V VM running Kernel 5.11 (Ubuntu 20.04) on a fast SSD. Mac - macOS 11.5.1 running on a Mac mini on a 1TB Apple SSD. Windows - Same host as Linux, a preview version of Windows 11. This number is from a patch later in the series. Adding 500 files to the repo with 'git add' Times reported in seconds. core.fsyncObjectFiles | Linux | Mac | Windows ----------------------|-------|-------|-------- false | 0.06 | 0.35 | 0.61 true | 1.88 | 11.18 | 2.47 batch | 0.15 | 0.41 | 1.53 Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 26 ++++++++++--- Makefile | 6 +++ builtin/add.c | 3 +- bulk-checkin.c | 70 ++++++++++++++++++++++++++++++++++- bulk-checkin.h | 4 +- cache.h | 8 +++- config.c | 8 +++- config.mak.uname | 2 + configure.ac | 8 ++++ environment.c | 2 +- git-compat-util.h | 7 ++++ object-file.c | 12 +----- wrapper.c | 36 ++++++++++++++++++ write-or-die.c | 2 +- 14 files changed, 170 insertions(+), 24 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..0006d90980d 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -548,12 +548,26 @@ core.whitespace:: errors. The default tab width is 8. Allowed values are 1 to 63. core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. -+ -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). + A value indicating the level of effort Git will expend in + trying to make objects added to the repo durable in the event + of an unclean system shutdown. This setting currently only + controls the object store, so updates to any refs or the + index may not be equally durable. ++ +* `false` allows data to remain in file system caches according to + operating system policy, whence it may be lost if the system loses power + or crashes. +* `true` triggers a data integrity flush for each object added to the + object store. This is the safest setting that is likely to ensure durability + across all operating systems and file systems that honor the 'fsync' system + call. However, this setting comes with a significant performance cost on + common hardware. +* `batch` enables an experimental mode that uses interfaces available in some + operating systems to write object data with a minimal set of FLUSH CACHE + (or equivalent) commands sent to the storage controller. If the operating + system interfaces are not available, this mode behaves the same as `true`. + This mode is expected to be safe on macOS for repos stored on HFS+ or APFS + filesystems and on Windows for repos stored on NTFS or ReFS. core.preloadIndex:: Enable parallel index preload for operations like 'git diff' diff --git a/Makefile b/Makefile index 9573190f1d7..143d30f04cb 100644 --- a/Makefile +++ b/Makefile @@ -406,6 +406,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1896,6 +1898,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/builtin/add.c b/builtin/add.c index 09e684585d9..c58dfcd4bc3 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -670,7 +670,8 @@ int cmd_add(int argc, const char **argv, const char *prefix) if (chmod_arg && pathspec.nr) exit_status |= chmod_pathspec(&pathspec, chmod_arg[0], show_only); - unplug_bulk_checkin(); + + unplug_bulk_checkin(&lock_file); finish: if (write_locked_index(&the_index, &lock_file, diff --git a/bulk-checkin.c b/bulk-checkin.c index f117d62c908..47b42f610c0 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -3,15 +3,19 @@ */ #include "cache.h" #include "bulk-checkin.h" +#include "lockfile.h" #include "repository.h" #include "csum-file.h" #include "pack.h" #include "strbuf.h" +#include "string-list.h" #include "packfile.h" #include "object-store.h" static int bulk_checkin_plugged; +static struct string_list bulk_fsync_state = STRING_LIST_INIT_DUP; + static struct bulk_checkin_state { char *pack_tmp_name; struct hashfile *f; @@ -62,6 +66,32 @@ clear_exit: reprepare_packed_git(the_repository); } +static void do_sync_and_rename(struct string_list *fsync_state, struct lock_file *lock_file) +{ + if (fsync_state->nr) { + struct string_list_item *rename; + + /* + * Issue a full hardware flush against the lock file to ensure + * that all objects are durable before any renames occur. + * The code in fsync_and_close_loose_object_bulk_checkin has + * already ensured that writeout has occurred, but it has not + * flushed any writeback cache in the storage hardware. + */ + fsync_or_die(get_lock_file_fd(lock_file), get_lock_file_path(lock_file)); + + for_each_string_list_item(rename, fsync_state) { + const char *src = rename->string; + const char *dst = rename->util; + + if (finalize_object_file(src, dst)) + die_errno(_("could not rename '%s' to '%s'"), src, dst); + } + + string_list_clear(fsync_state, 1); + } +} + static int already_written(struct bulk_checkin_state *state, struct object_id *oid) { int i; @@ -256,6 +286,42 @@ static int deflate_to_pack(struct bulk_checkin_state *state, return 0; } +static void add_rename_bulk_checkin(struct string_list *fsync_state, + const char *src, const char *dst) +{ + string_list_insert(fsync_state, src)->util = xstrdup(dst); +} + +int fsync_and_close_loose_object_bulk_checkin(int fd, const char *tmpfile, + const char *filename) +{ + if (fsync_object_files != FSYNC_OBJECT_FILES_OFF) { + /* + * If we have a plugged bulk checkin, we issue a call that + * cleans the filesystem page cache but avoids a hardware flush + * command. Later on we will issue a single hardware flush + * before renaming files as part of do_sync_and_rename. + */ + if (bulk_checkin_plugged && + fsync_object_files == FSYNC_OBJECT_FILES_BATCH && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) { + add_rename_bulk_checkin(&bulk_fsync_state, tmpfile, filename); + if (close(fd)) + die_errno(_("error when closing loose object file")); + + return 0; + + } else { + fsync_or_die(fd, "loose object file"); + } + } + + if (close(fd)) + die_errno(_("error when closing loose object file")); + + return finalize_object_file(tmpfile, filename); +} + int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags) @@ -273,10 +339,12 @@ void plug_bulk_checkin(void) bulk_checkin_plugged = 1; } -void unplug_bulk_checkin(void) +void unplug_bulk_checkin(struct lock_file *lock_file) { assert(bulk_checkin_plugged); bulk_checkin_plugged = 0; if (bulk_checkin_state.f) finish_bulk_checkin(&bulk_checkin_state); + + do_sync_and_rename(&bulk_fsync_state, lock_file); } diff --git a/bulk-checkin.h b/bulk-checkin.h index b26f3dc3b74..8efb01ed669 100644 --- a/bulk-checkin.h +++ b/bulk-checkin.h @@ -6,11 +6,13 @@ #include "cache.h" +int fsync_and_close_loose_object_bulk_checkin(int fd, const char *tmpfile, const char *filename); + int index_bulk_checkin(struct object_id *oid, int fd, size_t size, enum object_type type, const char *path, unsigned flags); void plug_bulk_checkin(void); -void unplug_bulk_checkin(void); +void unplug_bulk_checkin(struct lock_file *); #endif diff --git a/cache.h b/cache.h index bd4869beee4..cde6c6ae6b1 100644 --- a/cache.h +++ b/cache.h @@ -985,7 +985,13 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; -extern int fsync_object_files; +enum FSYNC_OBJECT_FILES_MODE { + FSYNC_OBJECT_FILES_OFF, + FSYNC_OBJECT_FILES_ON, + FSYNC_OBJECT_FILES_BATCH +}; + +extern enum FSYNC_OBJECT_FILES_MODE fsync_object_files; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/config.c b/config.c index f33abeab851..ab1980f8fec 100644 --- a/config.c +++ b/config.c @@ -1509,7 +1509,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { - fsync_object_files = git_config_bool(var, value); + if (!value) + return config_error_nonbool(var); + if (!strcasecmp(value, "batch")) + fsync_object_files = FSYNC_OBJECT_FILES_BATCH; + else + fsync_object_files = git_config_bool(var, value) + ? FSYNC_OBJECT_FILES_ON : FSYNC_OBJECT_FILES_OFF; return 0; } diff --git a/config.mak.uname b/config.mak.uname index 69413fb3dc0..8c07f2265a8 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -53,6 +53,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease SANE_TEXT_GREP=-a FREAD_READS_DIRECTORIES = UnfortunatelyYes @@ -133,6 +134,7 @@ ifeq ($(uname_S),Darwin) COMPAT_OBJS += compat/precompose_utf8.o BASIC_CFLAGS += -DPRECOMPOSE_UNICODE BASIC_CFLAGS += -DPROTECT_HFS_DEFAULT=1 + BASIC_CFLAGS += -DFSYNC_DOESNT_FLUSH=1 HAVE_BSD_SYSCTL = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes HAVE_NS_GET_EXECUTABLE_PATH = YesPlease diff --git a/configure.ac b/configure.ac index 031e8d3fee8..c711037d625 100644 --- a/configure.ac +++ b/configure.ac @@ -1090,6 +1090,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/environment.c b/environment.c index d6b22ede7ea..3e23eafff80 100644 --- a/environment.c +++ b/environment.c @@ -43,7 +43,7 @@ const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int core_compression_level; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; +enum FSYNC_OBJECT_FILES_MODE fsync_object_files; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index b46605300ab..d14e2436276 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1210,6 +1210,13 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/object-file.c b/object-file.c index 5421811273e..94a63809613 100644 --- a/object-file.c +++ b/object-file.c @@ -1859,16 +1859,6 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, return 0; } -/* Finalize a file on disk, and close it. */ -static int close_loose_object(int fd, const char *tmpfile, const char *filename) -{ - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); - if (close(fd) != 0) - die_errno(_("error when closing loose object file")); - return finalize_object_file(tmpfile, filename); -} - /* Size of directory component, including the ending '/' */ static inline int directory_size(const char *filename) { @@ -1982,7 +1972,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr, warning_errno(_("failed futimes() on %s"), tmp_file.buf); } - return close_loose_object(fd, tmp_file.buf, filename.buf); + return fsync_and_close_loose_object_bulk_checkin(fd, tmp_file.buf, filename.buf); } static int freshen_loose_object(const struct object_id *oid) diff --git a/wrapper.c b/wrapper.c index 563ad590df1..37a8b61a7df 100644 --- a/wrapper.c +++ b/wrapper.c @@ -538,6 +538,42 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +int git_fsync(int fd, enum fsync_action action) +{ + if (action == FSYNC_WRITEOUT_ONLY) { +#ifdef __APPLE__ + /* + * on Mac OS X, fsync just causes filesystem cache writeback but does not + * flush hardware caches. + */ + return fsync(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to issue + * a writeback without a hardware flush. An offset of 0 and size of 0 + * indicates writeout of the entire file and the wait flags ensure that all + * dirty data is written to the disk (potentially in a disk-side cache) + * before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + + errno = ENOSYS; + return -1; + } + +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); +#else + return fsync(fd); +#endif +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index d33e68f6abb..8f53953d4ab 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -57,7 +57,7 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) void fsync_or_die(int fd, const char *msg) { - while (fsync(fd) < 0) { + while (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) { if (errno != EINTR) die_errno("fsync error on '%s'", msg); } From patchwork Fri Aug 27 23:49:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12462977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45673C4320A for ; Fri, 27 Aug 2021 23:49:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2C33260F91 for ; Fri, 27 Aug 2021 23:49:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232689AbhH0Xuk (ORCPT ); Fri, 27 Aug 2021 19:50:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232616AbhH0Xub (ORCPT ); Fri, 27 Aug 2021 19:50:31 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 017ABC061796 for ; Fri, 27 Aug 2021 16:49:42 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id n5so12683169wro.12 for ; Fri, 27 Aug 2021 16:49:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WhvHe8govudP9XtIENQ0EqhwppcAIrn83E39aQbHy/I=; b=ib6u++jnH6uxHJEPr7vemGvNNBkstVYjNG4LJSoFBAsRKlEUcQ2xK8BY3IVzuus3le 4kheCqdmh6ZiFeG7jzRs0mHffOxzIH5huSE1tXRbmazpoqYBqORTdBFkKXazkVVHm39e uU+SLjVUtl4IhVhGP+5c6PWqL8TV4JB9ySmGYGPgBPgNrNjnG4O9DePksTWXYESHrj95 J5KmhA9BmWVzSuWN5cIMo+9JYRNI9+vi0ZhDsuoElp/n3IzpGPTE03anfDSlE+LErxNJ LOLZD06xale4JQxM7pZuMGk169/+xjR2ABS2/EIR/n5URzKYPcVotB8b7QsgkPERzsob 3aTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WhvHe8govudP9XtIENQ0EqhwppcAIrn83E39aQbHy/I=; b=Q/VgkvmuRcfju7drEk5n6O9vV3VYjylsCIdI1+lGZCKBa/VWgDwCITLRw5QV6Oatys 7CTxTxTCjZgZ4+q9iuSPz/9GFf3Dx9xQ15xnTJNG3VJAn7Sp88zRwCWyrW/1HoYhOEfr Hg4avqd/5cS2CBHppmQYcAXkFRFEiH6KjxOj6N2xkjmcKHku3GRPKb2Ps6XuytMDvZqM PZade0gPqmDAoiw6NjF2i/NdPq8+R5SMNT684IqZm/Oh9ZFtZaXxKJAorH10T3YIrtob aoVRkgIXAGO11xaTqnQoEo43tUCjuvgiS2h7dADRZkshkiUZvjH2B3mSe5S7jM5QF1QZ +piw== X-Gm-Message-State: AOAM532pjd+EA1ziCy7O1YHuAW6737aaCrNQcGsxxTsOwN3uRQlxMnPr l5IkK/2XcuQ1eoO3FYA7A+lQGTej2aU= X-Google-Smtp-Source: ABdhPJy8z8cm1KR2QKnCSfXBu2QekNHH+Cg9P/sy8E/kn+JAAXmiMo7h80U81/D5jr09yjilFd6kVA== X-Received: by 2002:a5d:63ca:: with SMTP id c10mr13370936wrw.16.1630108180685; Fri, 27 Aug 2021 16:49:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q13sm7598981wrv.79.2021.08.27.16.49.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Aug 2021 16:49:40 -0700 (PDT) Message-Id: <546ad9c82e8e0c2eb4683f9f360d8f30e2136020.1630108177.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 27 Aug 2021 23:49:35 +0000 Subject: [PATCH v2 4/6] core.fsyncobjectfiles: add windows support for batch mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Neeraj-Personal , Johannes Schindelin , Jeff King , Jeff Hostetler , Christoph Hellwig , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit adds a win32 implementation for fsync_no_flush that is called git_fsync. The 'NtFlushBuffersFileEx' function being called is available since Windows 8. If the function is not available, we return -1 and Git falls back to doing a full fsync. The operating system is told to flush data only without a hardware flush primitive. A later full fsync will cause the metadata log to be flushed and then the disk cache to be flushed on NTFS and ReFS. Other filesystems will treat this as a full flush operation. I added a new file here for this system call so as not to conflict with downstream changes in the git-for-windows repository related to fscache. Signed-off-by: Neeraj Singh --- compat/mingw.h | 3 +++ compat/win32/flush.c | 29 +++++++++++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 ++- wrapper.c | 4 ++++ 5 files changed, 40 insertions(+), 1 deletion(-) create mode 100644 compat/win32/flush.c diff --git a/compat/mingw.h b/compat/mingw.h index 87944dfec72..b5c950f1e30 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..c013920ce37 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,29 @@ +#include "../../git-compat-util.h" +#include +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + /* See https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-ntflushbuffersfileex */ + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.mak.uname b/config.mak.uname index 8c07f2265a8..ef1fd109b74 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -450,6 +450,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -624,6 +625,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 171b4124afe..b573a5ee122 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,7 +261,8 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c + list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c + compat/win32/flush.c compat/win32/path-utils.c compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c compat/win32/trace2_win32_process_info.c compat/win32/dirent.c compat/nedmalloc/nedmalloc.c compat/strdup.c) diff --git a/wrapper.c b/wrapper.c index 37a8b61a7df..d951306b33e 100644 --- a/wrapper.c +++ b/wrapper.c @@ -563,6 +563,10 @@ int git_fsync(int fd, enum fsync_action action) SYNC_FILE_RANGE_WAIT_AFTER); #endif +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + errno = ENOSYS; return -1; } From patchwork Fri Aug 27 23:49:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12462971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20016C43214 for ; Fri, 27 Aug 2021 23:49:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0242160F4F for ; Fri, 27 Aug 2021 23:49:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232643AbhH0Xug (ORCPT ); Fri, 27 Aug 2021 19:50:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232619AbhH0Xuc (ORCPT ); Fri, 27 Aug 2021 19:50:32 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 911C9C06179A for ; Fri, 27 Aug 2021 16:49:42 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id c8-20020a7bc008000000b002e6e462e95fso10311325wmb.2 for ; Fri, 27 Aug 2021 16:49:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IXJxACXefm+q8uf4n4bu/C6KwsEnKUjBPd8VAtER6Oc=; b=Y4Aoz2ac57UyaxxewCytR9cbGyA9sc9Q5rwnfWO+6ryfQMwrr1UOizbdyUmLv/8xfr MiuaCuJH3X68J0mZh+6pL6f+YK80jjxcQKutq6F1S4kJdDs0v5Gz+BQK0djE1bEMBe6s 0Q43S7NaQnmPTWT1tgUzTl7M4Nvv5RDHtYmf17Ni5HzbkF9waM7rxj+5aQjpTehqc00B wg0LcQKdSgS4ixl0xSQRYCzdJmyLJ+YWtW8faznq2VObzBWByEiWJM2qRd4rk4twVxJo ehvbcIzNg4YR6We/jPEIlYMGxKOyjvkZGJrItR2IlX2VWlv30wVJ6RPCZnH+8aewlEKK 2CwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IXJxACXefm+q8uf4n4bu/C6KwsEnKUjBPd8VAtER6Oc=; b=cT8acz99gdk0/727Q4ygMrYXxjUc828RDyTM2K2eY2QPP4qUwMqj7RstpefUOUAUQ2 R5lndQhCCAYXazLY9GnaQBF9UC6hbXbPObyx4pJ++GqCPKxajn+I1C/WW++L8hGr8pot 0orwa8X8h4CnIXPkUudYqhwcMvZ8IB27h4XldtxAWyWlteruuG5Gpb1CKdltdZHcEsMi Jk2vcpBUI6+42Vq0V790Ej4+IVRI5ezVWupNL4uzzrvvNZO9pm/XBfs2HbvhmM5NuPvt X8Q61pSmZnu5jQenUsvc0CQ3YotMrF69oZywmN169x0cX168/QjRq2Gy39asMoTuOGvs efjA== X-Gm-Message-State: AOAM533D6gz/lNhV3QgiNnVqRoC2NJCwKA1spWhSNBYzjhR3olDlfCA3 s9eG7bYqc3Y+FCvWEpItPr1FU/PzzpM= X-Google-Smtp-Source: ABdhPJxRjjnsjbBxyR2dWgP/OmR04Usw4JqIg1iYXnftMaJfjPYr75GVyul0dCJzg8l7VTmlDsAhhA== X-Received: by 2002:a1c:3b05:: with SMTP id i5mr11103981wma.136.1630108181253; Fri, 27 Aug 2021 16:49:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 129sm6719367wmz.26.2021.08.27.16.49.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Aug 2021 16:49:40 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 27 Aug 2021 23:49:36 +0000 Subject: [PATCH v2 5/6] update-index: use the bulk-checkin infrastructure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Neeraj-Personal , Johannes Schindelin , Jeff King , Jeff Hostetler , Christoph Hellwig , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh The update-index functionality is used internally by 'git stash push' to setup the internal stashed commit. This change enables bulk-checkin for update-index infrastructure to speed up adding new objects to the object database by leveraging the pack functionality and the new bulk-fsync functionality. This mode is enabled when passing paths to update-index via the --stdin flag, as is done by 'git stash'. There is some risk with this change, since under batch fsync, the object files will not be available until the update-index is entirely complete. This usage is unlikely, since any tool invoking update-index and expecting to see objects would have to snoop the output of --verbose to find out when update-index has actually processed a given path. Additionally the index is locked for the duration of the update. Signed-off-by: Neeraj Singh --- builtin/update-index.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/builtin/update-index.c b/builtin/update-index.c index f1f16f2de52..64d025cf49e 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -5,6 +5,7 @@ */ #define USE_THE_INDEX_COMPATIBILITY_MACROS #include "cache.h" +#include "bulk-checkin.h" #include "config.h" #include "lockfile.h" #include "quote.h" @@ -1152,6 +1153,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) struct strbuf unquoted = STRBUF_INIT; setup_work_tree(); + plug_bulk_checkin(); while (getline_fn(&buf, stdin) != EOF) { char *p; if (!nul_term_line && buf.buf[0] == '"') { @@ -1166,6 +1168,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) chmod_path(set_executable_bit, p); free(p); } + unplug_bulk_checkin(&lock_file); strbuf_release(&unquoted); strbuf_release(&buf); } From patchwork Fri Aug 27 23:49:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12462973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9D75C432BE for ; Fri, 27 Aug 2021 23:49:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BB28A60EBA for ; Fri, 27 Aug 2021 23:49:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232655AbhH0Xui (ORCPT ); Fri, 27 Aug 2021 19:50:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232625AbhH0Xuc (ORCPT ); Fri, 27 Aug 2021 19:50:32 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3521FC0617A8 for ; Fri, 27 Aug 2021 16:49:43 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id u9so12720228wrg.8 for ; Fri, 27 Aug 2021 16:49:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Vrb0Jnn5ImSQtleFZTqPwnJj+nFf4VQvZpqsrSdQ7Xc=; b=gic9i+lb7ojJT7ZZ8akkE1I3PL4OwIOPclcBkcGN5KGuRr1CrLnawlYA5WoeT+TQlp EcdbPRFEjtECK2e0Br/VfZcP7TXH6ITS9N7ni57Fr36ZrU2xXd2h0HVq/7KGvNLr7SkZ PeTlWvTd/d+ME4RTXdNZmJMGbDlSsZwlW/0hRyX05x7x6+1qYbfv9EWMnaepXIZ7Pp+P to/JVPPZdNYOpzwsNbMagEEKIjEhfLDPtviQtbS5aYGztP8j32XdNi3KHrHFV1G/Iq35 LE0VSiDOT0yaueSbO1p44dGx0JIoUBXzU5lcA6NH9fEU3EJxPvVGTGF/7XpxfvMiOnbs viJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Vrb0Jnn5ImSQtleFZTqPwnJj+nFf4VQvZpqsrSdQ7Xc=; b=IUyVkeV9jTVeaDoHQnctGOqYalC9x9kS18adqhKtOSQaJy8HaOqU9Ef2W07umL2Nzh mYa87Puq0qqu0DnUBdmglGvWnQd65HOazPCNToPdQD9a7pjD+MFkU3d1lIcUIHEQ1Wgo AtSWVzzY8aFBOUUrafYzBLyXMA75fmhI7xgHa8Ny/IxvkSp8W/7qcnpmaVzBh0GEqs66 5RVcZI6Nfn3GXEp9RyciiPJAC4g7RpE0YNZdWsqS6kfdwAUQdrw3ne3zRmwV2sxCof5m hw0I2wJrMAzkUWWnUshc6xmli5eTAiE5ge0MWdiDESyujUz2lEwautX3nUOdSooe/lyO UiDA== X-Gm-Message-State: AOAM530JFVr2/Q6U9CONAefoh0w1nPiY474icXnKTWKRfY8Nr5SdRR5c MZ6g/YNoIO43WN7tO9WekhKEfZIEAGE= X-Google-Smtp-Source: ABdhPJymuqcMtyKIzdKyfUraRl7PvBgdqrirPsMiZBHgI2hNdWEng9M1qgJBN0EtD+YB6VtgKQ213A== X-Received: by 2002:adf:b35c:: with SMTP id k28mr12898671wrd.169.1630108181771; Fri, 27 Aug 2021 16:49:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o12sm7476313wmr.2.2021.08.27.16.49.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Aug 2021 16:49:41 -0700 (PDT) Message-Id: <73b5d41be94d6e5571cbe5b8dd0d0f74edc4b474.1630108177.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 27 Aug 2021 23:49:37 +0000 Subject: [PATCH v2 6/6] core.fsyncobjectfiles: performance tests for add and stash Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Neeraj-Personal , Johannes Schindelin , Jeff King , Jeff Hostetler , Christoph Hellwig , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Add a basic performance test for "git add" and "git stash" of a lot of new objects with various fsync settings. Signed-off-by: Neeraj Singh --- t/perf/lib-unique-files.sh | 32 ++++++++++++++++++++++++++ t/perf/p3700-add.sh | 43 +++++++++++++++++++++++++++++++++++ t/perf/p3900-stash.sh | 46 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 121 insertions(+) create mode 100644 t/perf/lib-unique-files.sh create mode 100755 t/perf/p3700-add.sh create mode 100755 t/perf/p3900-stash.sh diff --git a/t/perf/lib-unique-files.sh b/t/perf/lib-unique-files.sh new file mode 100644 index 00000000000..10083395ae5 --- /dev/null +++ b/t/perf/lib-unique-files.sh @@ -0,0 +1,32 @@ +# Helper to create files with unique contents + +test_create_unique_files_base__=$(date -u) +test_create_unique_files_counter__=0 + +# Create multiple files with unique contents. Takes the number of +# directories, the number of files in each directory, and the base +# directory. +# +# test_create_unique_files 2 3 . -- Creates 2 directories with 3 files +# each in the current directory, all +# with unique contents. + +test_create_unique_files() { + test "$#" -ne 3 && BUG "3 param" + + local dirs=$1 + local files=$2 + local basedir=$3 + + for i in $(test_seq $dirs) + do + local dir=$basedir/dir$i + + mkdir -p "$dir" > /dev/null + for j in $(test_seq $files) + do + test_create_unique_files_counter__=$((test_create_unique_files_counter__ + 1)) + echo "$test_create_unique_files_base__.$test_create_unique_files_counter__" >"$dir/file$j.txt" + done + done +} diff --git a/t/perf/p3700-add.sh b/t/perf/p3700-add.sh new file mode 100755 index 00000000000..4ca3224f364 --- /dev/null +++ b/t/perf/p3700-add.sh @@ -0,0 +1,43 @@ +#!/bin/sh +# +# This test measures the performance of adding new files to the object database +# and index. The test was originally added to measure the effect of the +# core.fsyncObjectFiles=batch mode, which is why we are testing different values +# of that setting explicitly and creating a lot of unique objects. + +test_description="Tests performance of add" + +. ./perf-lib.sh + +. $TEST_DIRECTORY/perf/lib-unique-files.sh + +test_perf_default_repo +test_checkout_worktree + +dir_count=10 +files_per_dir=50 +total_files=$((dir_count * files_per_dir)) + +# We need to create the files each time we run the perf test, but +# we do not want to measure the cost of creating the files, so run +# the tet once. +if test "$GIT_PERF_REPEAT_COUNT" -ne 1 +then + echo "warning: Setting GIT_PERF_REPEAT_COUNT=1" >&2 + GIT_PERF_REPEAT_COUNT=1 +fi + +for m in false true batch +do + test_expect_success "create the files for core.fsyncObjectFiles=$m" ' + git reset --hard && + # create files across directories + test_create_unique_files $dir_count $files_per_dir files + ' + + test_perf "add $total_files files (core.fsyncObjectFiles=$m)" " + git -c core.fsyncobjectfiles=$m add files + " +done + +test_done diff --git a/t/perf/p3900-stash.sh b/t/perf/p3900-stash.sh new file mode 100755 index 00000000000..407b95c104b --- /dev/null +++ b/t/perf/p3900-stash.sh @@ -0,0 +1,46 @@ +#!/bin/sh +# +# This test measures the performance of adding new files to the object database +# and index. The test was originally added to measure the effect of the +# core.fsyncObjectFiles=batch mode, which is why we are testing different values +# of that setting explicitly and creating a lot of unique objects. + +test_description="Tests performance of stash" + +. ./perf-lib.sh + +. $TEST_DIRECTORY/perf/lib-unique-files.sh + +test_perf_default_repo +test_checkout_worktree + +dir_count=10 +files_per_dir=50 +total_files=$((dir_count * files_per_dir)) + +# We need to create the files each time we run the perf test, but +# we do not want to measure the cost of creating the files, so run +# the tet once. +if test "$GIT_PERF_REPEAT_COUNT" -ne 1 +then + echo "warning: Setting GIT_PERF_REPEAT_COUNT=1" >&2 + GIT_PERF_REPEAT_COUNT=1 +fi + +for m in false true batch +do + test_expect_success "create the files for core.fsyncObjectFiles=$m" ' + git reset --hard && + # create files across directories + test_create_unique_files $dir_count $files_per_dir files + ' + + # We only stash files in the 'files' subdirectory since + # the perf test infrastructure creates files in the + # current working directory that need to be preserved + test_perf "stash 500 files (core.fsyncObjectFiles=$m)" " + git -c core.fsyncobjectfiles=$m stash push -u -- files + " +done + +test_done