From patchwork Wed Mar 9 23:03:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12775695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB521C433F5 for ; Wed, 9 Mar 2022 23:03:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235074AbiCIXEX (ORCPT ); Wed, 9 Mar 2022 18:04:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229940AbiCIXEW (ORCPT ); Wed, 9 Mar 2022 18:04:22 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB63ED95E0 for ; Wed, 9 Mar 2022 15:03:22 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id r6so4849013wrr.2 for ; Wed, 09 Mar 2022 15:03:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=X4CTYWWdy5iVycp7d4sxXVOlKFY9zfnu5srEBSA4A74=; b=Chewpo51IM+3vxP/+hXq1j/XTqayTe88sAXS10oaT+9zF+1jPUITFg4G0d+RWwTcP0 dG95Ki6Gckup/mlO6ka6F/I6QNtfzThghlT9wEWwB3YPfzeMljhbySknHF8NQd+RrkW6 dN2q9pV/l9OqV8BnuOTd7hn6juLm200ZXa9u3GGPe1N/BgInreIHipPIeuwqNiyUJkxE 5B+WJOy+eIcZNyV5X4z+Kp8VYfKGixP5q8rMtOxNzSgQ8feDq+HcFpayl0uWueqbvrpH V5hSj4QChLhDDxTaBK/7BEsRPJX8AFrFHbvIo7Dsf4UJ2TFVLXhGFsYS0eehzi+D1ELq bbHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=X4CTYWWdy5iVycp7d4sxXVOlKFY9zfnu5srEBSA4A74=; b=RB74/zAGbtCJF4GBhdGRYMGyXkgt/FHlf3/CNkQnEt5TwM3tgVL284YUyl0DcEX4JE rCM7747USLBt14JtjOVmOs1u3JEXrjPYr1DCcwOyi5io6+KicHLErHu6nGxoh4xTvwcU kcKZ74yPFLO4mBnIaZj/RwpbLX1z98etoMqCm3e3wLX9xTEwOmmi0VaO4sRr1EMKrArq 8Y5mDtv0aaqQ1lpDRPnK9Nj3USHunFSjQG4mhlA8K9dsRKYpohGlhFP/wz9dfqXCGZCL +9RwRcDsIqZEI+hAGdc32d7tmPNzXm/o0mECueCvtn3AAYrvQ9xwBNybY+9dpxAx0CEH dn2g== X-Gm-Message-State: AOAM533v6Nx8AXQxWxZoBuZtFlAR7GUGM8avQh7eHZehCWwxvetttc5Y JdwKvWcq0lvowUPVRrJH7mwLywquiZo= X-Google-Smtp-Source: ABdhPJz4PPvZV9c96aOfyTkpplh+7vYmBSbb0C/ZNDqB/ZWR9jxTQWqzebFkmi0uaiqzgQtNDknn/g== X-Received: by 2002:adf:eb4d:0:b0:1ed:c1f7:a951 with SMTP id u13-20020adfeb4d000000b001edc1f7a951mr1317148wrn.454.1646867001162; Wed, 09 Mar 2022 15:03:21 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c2-20020a056000184200b002037b40de23sm2225809wri.8.2022.03.09.15.03.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Mar 2022 15:03:20 -0800 (PST) Message-Id: <685b1db888079c83573cfd984ae64f46284544af.1646866998.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 09 Mar 2022 23:03:14 +0000 Subject: [PATCH v5 1/5] wrapper: move inclusion of CSPRNG headers the wrapper.c file Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Including NTSecAPI.h in git-compat-util.h causes build errors in any other file that includes winternl.h. That file was included in order to get access to the RtlGenRandom cryptographically secure PRNG. This change scopes the inclusion of all PRNG headers to just the wrapper.c file, which is the only place it is really needed. Signed-off-by: Neeraj Singh --- compat/winansi.c | 5 ----- git-compat-util.h | 12 ------------ wrapper.c | 14 ++++++++++++++ 3 files changed, 14 insertions(+), 17 deletions(-) diff --git a/compat/winansi.c b/compat/winansi.c index 936a80a5f00..3abe8dd5a27 100644 --- a/compat/winansi.c +++ b/compat/winansi.c @@ -4,11 +4,6 @@ #undef NOGDI -/* - * Including the appropriate header file for RtlGenRandom causes MSVC to see a - * redefinition of types in an incompatible way when including headers below. - */ -#undef HAVE_RTLGENRANDOM #include "../git-compat-util.h" #include #include diff --git a/git-compat-util.h b/git-compat-util.h index 876907b9df4..a25ebb822ee 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -197,12 +197,6 @@ #endif #include #define GIT_WINDOWS_NATIVE -#ifdef HAVE_RTLGENRANDOM -/* This is required to get access to RtlGenRandom. */ -#define SystemFunction036 NTAPI SystemFunction036 -#include -#undef SystemFunction036 -#endif #endif #include @@ -273,12 +267,6 @@ #else #include #endif -#ifdef HAVE_ARC4RANDOM_LIBBSD -#include -#endif -#ifdef HAVE_GETRANDOM -#include -#endif #ifdef NO_INTPTR_T /* * On I16LP32, ILP32 and LP64 "long" is the safe bet, however diff --git a/wrapper.c b/wrapper.c index 3258cdb171f..2a1aade473b 100644 --- a/wrapper.c +++ b/wrapper.c @@ -4,6 +4,20 @@ #include "cache.h" #include "config.h" +#ifdef HAVE_RTLGENRANDOM +/* This is required to get access to RtlGenRandom. */ +#define SystemFunction036 NTAPI SystemFunction036 +#include +#undef SystemFunction036 +#endif + +#ifdef HAVE_ARC4RANDOM_LIBBSD +#include +#endif +#ifdef HAVE_GETRANDOM +#include +#endif + static int memory_limit_check(size_t size, int gentle) { static size_t limit = 0; From patchwork Wed Mar 9 23:03:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12775696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AE1BC433F5 for ; Wed, 9 Mar 2022 23:03:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237480AbiCIXE1 (ORCPT ); Wed, 9 Mar 2022 18:04:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235778AbiCIXEZ (ORCPT ); Wed, 9 Mar 2022 18:04:25 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01868FBA6B for ; Wed, 9 Mar 2022 15:03:24 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id k29-20020a05600c1c9d00b003817fdc0f00so2343457wms.4 for ; Wed, 09 Mar 2022 15:03:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HBPTjOv4OhvSSNPIxoRm+JA0I5cKev3YRBzASTi95bg=; b=hDLE5TZ0zcOhVw9jkHqHHLkrokvmdhTOuTQlArb+R5ockZSiC9jnvjeEn49k6yviBw F4+4H7LMqOfz50V0MsKqaWfxlS7FjfZYINh9jbbCE6dC4WIH9p1UibTE2HkhY/Lk1SnA sKmXxvX7GA6Y65ANLGEOzRO3hj44oVqOGNCMrS8ji93JVYUAOkevgs3K4KfSSQytvcQM zhG0dhlp+f+6mKoO79Nyqg8IK5XmGsdJNLWgjy7eIgOURTkbaeEgUyv927wsqbQwe+pq AWFQhTPA+RIffyJcF7vADppZthyIJ+tGkSr0in2mrSogwwQXiZDInGV+bgMH99+ilNoG +vMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HBPTjOv4OhvSSNPIxoRm+JA0I5cKev3YRBzASTi95bg=; b=4dfMNpXGkw4FV3MvBT6fJVXaQujqW0pXaaJG8zNef1r2Pt4VgnUDm2jcDQluc5wIHn UWarvsZ1f/YEWB0k6BkOSy207aEJ+/R7CpzUHPaajhvjtbDRyPJw6f6iHR8iRppnqteZ gtMsG8dUYfXMCETeGWNglY38isNqFLEP1RF0hFFff4FTnpglWbCNM0ThotF8u/hhUUIQ ithq+9deMtBrpNn5IZVNKzgYc4hO5ZWVxdA3PzE4uMrvCe+0BVCwGibGvAoaExfKhfrr yBNOWqPLDZt9+Vr5QMZURt27MEVy/j4bs82P7zOvhbj/K/L6KoJo2XTQ+wvTi4T5d33B 6kLw== X-Gm-Message-State: AOAM530h5exEIKI/9lMYv0XzTiHECK/jOHKMG63CeQYcs1gr2XMbzsqD hv2kmFbmdD1YckUao1RycytwITEeh6Q= X-Google-Smtp-Source: ABdhPJxaG/NRXv59UjJ7uaiQ50rrr0yKDFWbxAZJttzKd3VuoJjKXNhIz7XoC+DtXu+mdnhfj0qTmQ== X-Received: by 2002:a1c:6a1a:0:b0:387:5ee7:750e with SMTP id f26-20020a1c6a1a000000b003875ee7750emr1275330wmc.85.1646867002205; Wed, 09 Mar 2022 15:03:22 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k13-20020a7bc40d000000b00381890032dfsm6030163wmi.1.2022.03.09.15.03.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Mar 2022 15:03:21 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 09 Mar 2022 23:03:15 +0000 Subject: [PATCH v5 2/5] core.fsyncmethod: add writeout-only mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 9 ++++ Makefile | 6 +++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++ config.c | 12 ++++++ config.mak.uname | 3 ++ configure.ac | 8 ++++ contrib/buildsystems/CMakeLists.txt | 16 ++++++-- environment.c | 1 + git-compat-util.h | 24 +++++++++++ wrapper.c | 64 +++++++++++++++++++++++++++++ write-or-die.c | 11 +++-- 13 files changed, 184 insertions(+), 8 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..dbb134f7136 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues pagecache writeback requests, but depending on the + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/Makefile b/Makefile index 6f0b4b775fe..17fd9b023a4 100644 --- a/Makefile +++ b/Makefile @@ -411,6 +411,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1897,6 +1899,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/cache.h b/cache.h index 04d4d2db25c..82f0194a3dd 100644 --- a/cache.h +++ b/cache.h @@ -995,6 +995,13 @@ extern char *git_replace_ref_base; extern int fsync_object_files; extern int use_fsync; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..291f90ea940 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "git-compat-util.h" +#include +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index 383b1a4885b..f3ff80b01c9 100644 --- a/config.c +++ b/config.c @@ -1600,6 +1600,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index 4352ea39e9b..404fff5dd04 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -463,6 +464,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -640,6 +642,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index 5ee25ec95c8..6bd6bef1c44 100644 --- a/configure.ac +++ b/configure.ac @@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index e44232f85d3..3a9e6241660 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,10 +261,18 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c - compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c - compat/win32/trace2_win32_process_info.c compat/win32/dirent.c - compat/nedmalloc/nedmalloc.c compat/strdup.c) + list(APPEND compat_SOURCES + compat/mingw.c + compat/winansi.c + compat/win32/flush.c + compat/win32/path-utils.c + compat/win32/pthread.c + compat/win32mmap.c + compat/win32/syslog.c + compat/win32/trace2_win32_process_info.c + compat/win32/dirent.c + compat/nedmalloc/nedmalloc.c + compat/strdup.c) set(NO_UNIX_SOCKETS 1) elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") diff --git a/environment.c b/environment.c index fd0501e77a5..3e3620d759f 100644 --- a/environment.c +++ b/environment.c @@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; int fsync_object_files; int use_fsync = -1; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index a25ebb822ee..41b97027457 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1265,6 +1265,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 2a1aade473b..2c44f71da42 100644 --- a/wrapper.c +++ b/wrapper.c @@ -553,6 +553,70 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +/* + * Some platforms return EINTR from fsync. Since fsync is invoked in some + * cases by a wrapper that dies on failure, do not expose EINTR to callers. + */ +static int fsync_loop(int fd) +{ + int err; + + do { + err = fsync(fd); + } while (err < 0 && errno == EINTR); + return err; +} + +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * On macOS, fsync just causes filesystem cache writeback but + * does not flush hardware caches. + */ + return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to + * issue a writeback without a hardware flush. An offset of + * 0 and size of 0 indicates writeout of the entire file and the + * wait flags ensure that all dirty data is written to the disk + * (potentially in a disk-side cache) before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On macOS, a special fcntl is required to really flush the + * caches within the storage controller. As of this writing, + * this is a very expensive operation on Apple SSDs. + */ +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); +#else + return fsync_loop(fd); +#endif + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index a3d5784cec9..9faa5f9f563 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) return; - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) From patchwork Wed Mar 9 23:03:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12775697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43B55C433EF for ; Wed, 9 Mar 2022 23:03:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237691AbiCIXEd (ORCPT ); Wed, 9 Mar 2022 18:04:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236021AbiCIXEZ (ORCPT ); Wed, 9 Mar 2022 18:04:25 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CB1FD95E0 for ; Wed, 9 Mar 2022 15:03:25 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id m42-20020a05600c3b2a00b00382ab337e14so4333336wms.3 for ; Wed, 09 Mar 2022 15:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=de8LoPziZFiNt20GcHwiL/PGLorEbIXCRknp2J9N4A4=; b=Xh0f3Zb1BA8YYQsj46V+DfixNrEamBVV8E7dvjOzxkD6bmREPE79w4FvwHTA6BmDUx yIqCjy6syyTlmiy5OlN3FxioPDmPCufTu31z8OyGpH2L6S4QSvdTh7lTln4RCyEIN5TZ SU0Fx+7AifF8MIbyblWynAHRo9mndvn30zxYOAvv51X5zgsaj5tPQfmDetW5gE4OZylT khyylsEmRXE5YlHfMD/Mr0maBMOmsoeXklcP3Mvm3KQ4c06FLgzQCjaPmR4/PFZ0eFzY UHtuopEZ7f5p9kpb7J/3arAPkW246Jk4BXLYXEW9B3QyeNnuTtda5lKV3tR0hRuqjSJ+ 6uhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=de8LoPziZFiNt20GcHwiL/PGLorEbIXCRknp2J9N4A4=; b=dY5YCwWc3SV+JKb2P2tefbsUnIXJFuiwWA0np4rplfh7GbF82lHXOGs29Bfq5qXQyU NtPfTcwRUVSR5ILQLwt2yZVJjdRyvHd2pzCl2im8w5Mdwi3cJM5xn1PLf27z0sY5GuGZ 3oC+GypQQKBS+dlIp3xZS2sUW9/m+ceEpELF3JJvhpJHL0PoV1Dh34mLXrKAmLjHMZDp ZsJiZyoOpBmX7huPL+Tsy0Nk4h/zsCRllOGSbWAPAFyODprPsrsjs4gfXpa6FthV8Uj9 SM48ArKML+OWcS/SqSoUnFNQABE6D6KMjYI35pw0mekxxssuid7UO8ZLSboiDZ7zawZK 33Sg== X-Gm-Message-State: AOAM532fo8ZkIsgWiRb1w7uvGZwFETJ7OKLQFY4qfisHkMQFZKgIdkPz T2Kt203IuIjiu/hLHlOwZP/MySYgDNY= X-Google-Smtp-Source: ABdhPJz7Sz2QYw2BMSrGKLhlXFNwGay8hbXof9JHyYrKjf/igLCxWq/g+N07I1GGhDXKMM5Mp6JZcw== X-Received: by 2002:a7b:c11a:0:b0:389:a2be:2a5a with SMTP id w26-20020a7bc11a000000b00389a2be2a5amr1244410wmi.25.1646867003346; Wed, 09 Mar 2022 15:03:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z5-20020a05600c0a0500b0037bb8df81a2sm7610360wmp.13.2022.03.09.15.03.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Mar 2022 15:03:22 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 09 Mar 2022 23:03:16 +0000 Subject: [PATCH v5 3/5] core.fsync: introduce granular fsync control Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit introduces the `core.fsync` configuration knob which can be used to control how components of the repository are made durable on disk. This setting allows future extensibility of the list of syncable components: * We issue a warning rather than an error for unrecognized components, so new configs can be used with old Git versions. * We support negation, so users can choose one of the aggregate options and then remove components that they don't want. Aggregate options are defined in a later patch in this series. This also supports the common request of doing absolutely no fysncing with the `core.fsync=none` value, which is expected to make the test suite faster. Complete documentation for the new setting is included in a later patch in the series so that it can be reviewed in final form. Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 8 ---- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 +- builtin/pack-objects.c | 24 ++++++++---- bulk-checkin.c | 5 ++- cache.h | 30 +++++++++++++- commit-graph.c | 3 +- config.c | 73 ++++++++++++++++++++++++++++++++++- csum-file.c | 5 ++- csum-file.h | 3 +- environment.c | 2 +- midx.c | 3 +- object-file.c | 3 +- pack-bitmap-write.c | 3 +- pack-write.c | 13 ++++--- read-cache.c | 2 +- 16 files changed, 144 insertions(+), 39 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index dbb134f7136..74399072843 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -556,14 +556,6 @@ core.fsyncMethod:: filesystem and storage hardware, data added to the repository may not be durable in the event of a system crash. This is the default mode on macOS. -core.fsyncObjectFiles:: - This boolean will enable 'fsync()' when writing object files. -+ -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). - core.preloadIndex:: Enable parallel index preload for operations like 'git diff' + diff --git a/builtin/fast-import.c b/builtin/fast-import.c index b7105fcad9b..f2c036a8955 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -865,7 +865,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index c45273de3b1..c5f12f14df5 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1290,7 +1290,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1512,7 +1512,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 178e611f09d..c14fee8e99f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1199,16 +1199,26 @@ static void write_pack_file(void) display_progress(progress_state, written); } - /* - * Did we write the wrong # entries in the header? - * If so, rewrite it like in fast-import - */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + /* + * We never fsync when writing to stdout since we may + * not be writing to an actual pack file. For instance, + * the upload-pack code passes a pipe here. Calling + * fsync on a pipe results in unnecessary + * synchronization with the reader on some platforms. + */ + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + /* + * If we wrote the wrong number of entries in the + * header, rewrite it like in fast-import. + */ + + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..a2cf9dcbc8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 82f0194a3dd..a5eaa60a7a8 100644 --- a/cache.h +++ b/cache.h @@ -993,8 +993,27 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; -extern int fsync_object_files; -extern int use_fsync; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { + FSYNC_COMPONENT_NONE, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; enum fsync_method { FSYNC_METHOD_FSYNC, @@ -1002,6 +1021,7 @@ enum fsync_method { }; extern enum fsync_method fsync_method; +extern int use_fsync; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; @@ -1708,6 +1728,12 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); ssize_t pread_in_full(int fd, void *buf, size_t count, off_t offset); diff --git a/commit-graph.c b/commit-graph.c index 265c010122e..64897f57d9f 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/config.c b/config.c index f3ff80b01c9..51a35715642 100644 --- a/config.c +++ b/config.c @@ -1323,6 +1323,70 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_entry { + const char *name; + enum fsync_component component_bits; +} fsync_component_table[] = { + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ + enum fsync_component output = 0; + + if (!strcmp(string, "none")) + return FSYNC_COMPONENT_NONE; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_table); ++i) { + const struct fsync_component_entry *entry = &fsync_component_table[i]; + + if (strncmp(entry->name, string, len)) + continue; + + found = 1; + if (negated) + output &= ~entry->component_bits; + else + output |= entry->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("ignoring unknown core.fsync component '%s'"), component); + free(component); + } + + string = ep; + } + + return output; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1600,6 +1664,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1613,7 +1684,7 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { - fsync_object_files = git_config_bool(var, value); + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); return 0; } diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..59ef3398ca2 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum fsync_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..0d29f528fbc 100644 --- a/csum-file.h +++ b/csum-file.h @@ -1,6 +1,7 @@ #ifndef CSUM_FILE_H #define CSUM_FILE_H +#include "cache.h" #include "hash.h" struct progress; @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index 3e3620d759f..378424b9af5 100644 --- a/environment.c +++ b/environment.c @@ -42,9 +42,9 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 865170bad05..107365d2114 100644 --- a/midx.c +++ b/midx.c @@ -1438,7 +1438,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & MIDX_WRITE_REV_INDEX && diff --git a/object-file.c b/object-file.c index 03bd6a3baf3..c9de912faca 100644 --- a/object-file.c +++ b/object-file.c @@ -1850,8 +1850,7 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, static void close_loose_object(int fd) { if (!the_repository->objects->odb->will_destroy) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, "loose object file"); } if (close(fd) != 0) diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index cab3eaa2acd..cf681547f2e 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..51812cb1299 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index 79b9b99ebf7..df869691fd4 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3089,7 +3089,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; From patchwork Wed Mar 9 23:03:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12775699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACF29C433F5 for ; Wed, 9 Mar 2022 23:03:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237795AbiCIXEh (ORCPT ); Wed, 9 Mar 2022 18:04:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236022AbiCIXE0 (ORCPT ); Wed, 9 Mar 2022 18:04:26 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CADDD1066F6 for ; Wed, 9 Mar 2022 15:03:25 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id u10so5264364wra.9 for ; Wed, 09 Mar 2022 15:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cjxUR2mJtVxsOIx6Eab4jEeqQQy3k4BYtli53r3TQcI=; b=AZ/hqiMCE+uXbu36kDZuJT2RDqAoFurcajdiHORehZ6PzT1WiFCt02hTf0p4ZlvrCg 9OzjyAU3g0kk47fUmlItBFD1yK9vXIKlA8UFeWuZ1gdFAtgwk2sBSHKBldEOzVw4dxGh WjJu6Gm/Ws/pTIPQzGtLsPh9bVsRTCQ+GeB3qhK6BiLgLazLVzUe/rHodUeN8/ZyY0RI q11JAdZpM0b7OCP9XS+VT5zz1P7gAMw84ufO0lAaa/bAFZb2hv60o0Ww/8kR50B+GwUC xh4SePQ4zAakaEWaVeGNMwB38MtN+jcOZV1uLaRWN4pNYgY3n9dwh7ngiKelLFzXZ1p5 bUSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cjxUR2mJtVxsOIx6Eab4jEeqQQy3k4BYtli53r3TQcI=; b=zrUgdhHpgBuXi9/6YHArE/8oIrLYFfJYlkGi1ulC4btHbo2m7yVsm4g2RiFOSHObPF 1rvQUhk2Pm2EulpYItnwEnX/wUqDC64USbe6qpWqXi19CTK+DuOQEhD1ybBipO5FK/HW ObZSXGD7DRQ7v1YurUg1SW2BiKbs7bD72gNtF0dXxFOKB6KGKaNnYg+ltznCaTySqpMT az9Kdo4jl8T85NH6KKVXPlsqgWEeG4+CtxM+ni7ELXZgtsoRIAodq1B0CSbxHYb1vSo0 vRalzLXQrQ8ItdyPWiWx7YiFCbRkU5aEwSFRAP7n8BsruJLuiR7gw9fo+cNRRs3CowEP QQsw== X-Gm-Message-State: AOAM530ns/4ouUODg8KGAiHya864hhdSwu/w6K5MxmW2N7X2VX3f7S05 4012ajBrG29dd3hL3fvlsxKQpdkjLZ8= X-Google-Smtp-Source: ABdhPJxde5NaqodUGHv0vshV0nsAVnV2XYgc2P6q+cJLTVDw4gy7/HGqz5kWaZlqg4rvx5d0jQcgNg== X-Received: by 2002:a5d:64a6:0:b0:203:66a0:474d with SMTP id m6-20020a5d64a6000000b0020366a0474dmr1432236wrp.392.1646867004247; Wed, 09 Mar 2022 15:03:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o7-20020a5d6707000000b001f067c7b47fsm4115599wru.27.2022.03.09.15.03.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Mar 2022 15:03:23 -0800 (PST) Message-Id: <9da808ba743673bb0c40b127fed7dd34c87d232d.1646866998.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 09 Mar 2022 23:03:17 +0000 Subject: [PATCH v5 4/5] core.fsync: new option to harden the index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh --- cache.h | 1 + config.c | 1 + read-cache.c | 19 +++++++++++++------ 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/cache.h b/cache.h index a5eaa60a7a8..3fefe8f6f60 100644 --- a/cache.h +++ b/cache.h @@ -1004,6 +1004,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, + FSYNC_COMPONENT_INDEX = 1 << 4, }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ diff --git a/config.c b/config.c index 51a35715642..bb44f6f506d 100644 --- a/config.c +++ b/config.c @@ -1331,6 +1331,7 @@ static const struct fsync_component_entry { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) diff --git a/read-cache.c b/read-cache.c index df869691fd4..7683b679258 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2842,7 +2842,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2856,6 +2856,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3089,7 +3090,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3144,7 +3151,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3238,7 +3245,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3248,7 +3255,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3357,7 +3364,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) From patchwork Wed Mar 9 23:03:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12775700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69B6BC433FE for ; Wed, 9 Mar 2022 23:03:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237832AbiCIXEi (ORCPT ); Wed, 9 Mar 2022 18:04:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236839AbiCIXE1 (ORCPT ); Wed, 9 Mar 2022 18:04:27 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A446120F4D for ; Wed, 9 Mar 2022 15:03:27 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id e24so5250418wrc.10 for ; Wed, 09 Mar 2022 15:03:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=iS29yHd10pfhgSmUnfUcOUWlb5yXia0SwcOp6jW9HYE=; b=mXP+x/kq/5pdDiQj+9ZN5lqhvilJspnN6rCMDko26xvpfI2wIYrMeBmG+3IgumO7Xn Iy6uqMofVWyDUxXWMoAMUjs0jTgaTAh4UKwjCPQr/yRYLxfND0g4ujWXklkVJye1X5Nr 6bllshMN519LLzoNjJIv3ALfyvBd389d+hSzfAC2azNuJrzQFwGc/EaqFNMlWqEK514i LETCqWf14Y3p/SkxsKSoM/eN9QSJAl3zgeuMMVtMRQVhqpTe8CH0rTG7L+e/2C8H+q/F ZVr9swSpww1wMQZESfvbi6C9cNUIrF9NWhdJJNoxghhHIWFMQEr/rucwQwRyfXeoD4ty fLtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=iS29yHd10pfhgSmUnfUcOUWlb5yXia0SwcOp6jW9HYE=; b=FTkJus+Dn7JEb5QuLQPokFlzK/jWNrMzZymgfoO57uVLEiqCY5OCnJtrC4VbjCcTyK ash7lsMCxhd4xdDU/HSxNPrZGdFxMN8mn+HTUkoxfAjBdu7PzKiaEBjYHkqC3Y3ZOZcF MklOnzYaGzyIZQHtIKr0zz9dKFzA5x4uw4S3X5aR3MP/87tC5JMgo+43O1RnpUQUNY0a uUbeN2MCLZqxv4qTwrE8SuWOTJG+5I3Gs/0aT3ohJwbABhqVMW6SoPTEVtopbEc9TczQ kO+MyTSBsoxAjNGAq6bqye0jxFrKph4KYRkZ4A/Y1rVFtqWhGwoXJYyzTAMpzdZWhsYU xu7Q== X-Gm-Message-State: AOAM533++W8s6e4/d0A5/1Lzq6dNCLo9GGrTFCNz2KVhYNjZLG1I/3kD zuexcaehAKLK8J3RHVfVg1h0dZ7RXlw= X-Google-Smtp-Source: ABdhPJzYTpvi5OIglgQAEINRPrQ1ayZApCnPeM2bnshJxWfGuarQDrp/umIfIHKXAmCJNQ6U1w2dUg== X-Received: by 2002:a5d:62cd:0:b0:1f0:23d2:b38c with SMTP id o13-20020a5d62cd000000b001f023d2b38cmr1437604wrv.82.1646867005142; Wed, 09 Mar 2022 15:03:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bg18-20020a05600c3c9200b0037c2ef07493sm3515688wmb.3.2022.03.09.15.03.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Mar 2022 15:03:24 -0800 (PST) Message-Id: <2d71346b10e8beb3c44bdf8e06694e4dafc657d5.1646866998.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 09 Mar 2022 23:03:18 +0000 Subject: [PATCH v5 5/5] core.fsync: documentation and user-friendly aggregate options Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit adds aggregate options for the core.fsync setting that are more user-friendly. These options are specified in terms of 'levels of safety', indicating which Git operations are considered to be sync points for durability. The new documentation is also included here in its entirety for ease of review. Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 36 +++++++++++++++++++++++++++++++++++ cache.h | 23 +++++++++++++++++++--- config.c | 6 ++++++ 3 files changed, 62 insertions(+), 3 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 74399072843..973805e8a98 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,42 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of parts of the repository which should be + hardened via the core.fsyncMethod when created or modified. You can + disable hardening of any component by prefixing it with a '-'. Later + items take precedence over earlier ones in the comma-separated list. + For example, `core.fsync=all,-pack-metadata` means "harden everything + except pack metadata." Items that are not hardened may be lost in the + event of an unclean system shutdown. Unless you have special + requirements, it is recommended that you leave this option as default + or pick one of `committed`, `added`, or `all`. ++ +* `none` disables fsync completely. This value must be specified alone. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. +* `objects` is an aggregate option that is equivalent to + `loose-object,pack`. +* `derived-metadata` is an aggregate option that is equivalent to + `pack-metadata,commit-graph`. +* `default` is an aggregate option that is equivalent to + `objects,derived-metadata,-loose-object`. This mode is enabled by default. + It has good performance, but risks losing recent work if the system shuts + down uncleanly, since commits, trees, and blobs in loose-object form may be + lost. +* `committed` is an aggregate option that is currently equivalent to + `objects`. This mode sacrifices some performance to ensure that all work + that is committed to the repository with `git commit` or similar commands + is preserved. +* `added` is an aggregate option that is currently equivalent to + `committed,index`. This mode sacrifices additional performance to + ensure that the results of commands like `git add` and similar operations + are preserved. +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. diff --git a/cache.h b/cache.h index 3fefe8f6f60..833f0236e68 100644 --- a/cache.h +++ b/cache.h @@ -1007,9 +1007,26 @@ enum fsync_component { FSYNC_COMPONENT_INDEX = 1 << 4, }; -#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \ + FSYNC_COMPONENTS_DERIVED_METADATA | \ + ~FSYNC_COMPONENT_LOOSE_OBJECT) + +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) + +#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ + FSYNC_COMPONENT_INDEX) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index bb44f6f506d..3976ec74fd4 100644 --- a/config.c +++ b/config.c @@ -1332,6 +1332,12 @@ static const struct fsync_component_entry { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, + { "default", FSYNC_COMPONENTS_DEFAULT }, + { "committed", FSYNC_COMPONENTS_COMMITTED }, + { "added", FSYNC_COMPONENTS_ADDED }, + { "all", FSYNC_COMPONENTS_ALL }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) From patchwork Thu Mar 10 09:53:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12776111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F1FDC433F5 for ; Thu, 10 Mar 2022 09:53:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241229AbiCJJyz (ORCPT ); Thu, 10 Mar 2022 04:54:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241119AbiCJJyc (ORCPT ); Thu, 10 Mar 2022 04:54:32 -0500 Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B874E4D271 for ; Thu, 10 Mar 2022 01:53:21 -0800 (PST) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id C1D725C02AF; Thu, 10 Mar 2022 04:53:20 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 10 Mar 2022 04:53:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=Qz18bNmdtCINU2zBuC2il/3KZTMz5KiJ1Gr4ts PN1Ag=; b=r3st5/IEcjnXhVqfVX0Di+wLOzRDQEXvjv2JCtk1JW6Yq0Z6cTAHnJ MfTk+u5HPw1GSRYIeA3mHn3dwLN/8SJimzEw2AUGd4PaEd2n3zQ1yxRI5wcV4/KA 871zvcxD1j482MhJdYnvdEKbS2wMBskewFtf8M65XWKtX1CJWnQdWFX6KK+sJyaK Gda0cXSBd6kBoWY/Cm62U9kW6DiLjh5pbayLtvxUXVXVbVL0rCkz0UkXWnh1X6jA AslatHxKEwK6pr12a/IsyA4IlBeJw4Lwvaw9aSXZBFZvnI3rBpj0PQFv548fbnuP owgMhqwNWX6Tprehbo5J2bWR2V/8K/3w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=Qz18bNmdtCINU2zBu C2il/3KZTMz5KiJ1Gr4tsPN1Ag=; b=AR/LHDbWu85HKyRCqdZGlNUROL6uSbGCY C+RmFTAeoTarOukmzd5kofx6w31EdcqEwMkStljtrv33ToiaDMSIJopin/UfKCOc z5uu+HCCr1Czkh/WfqingIznKj/Gngwx14Yxbj7ygt3gdN/KZGeaiwfv3h7GJA3+ heHKNCWVdVTP4fFc9gwMGOQxlHVWppB3IgmAQmsoITYYIeVKdchMCAITGbJLeoxs yWoEF4hKG7l2+dgQiBu04EXrrSBTMJmpUQdzPDnx6CgSR4/DvWc2V3+hLtd+jnhm DvVmciVQ4GFf5vLMPqNd+MIsKNnSo9mUp9jpg6RlJO8ZsYhaKw5uQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddruddvtddgtdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Mar 2022 04:53:19 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 78d412dd (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Mar 2022 09:53:18 +0000 (UTC) Date: Thu, 10 Mar 2022 10:53:17 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Junio C Hamano Subject: [PATCH 6/8] core.fsync: add `fsync_component()` wrapper which doesn't die Message-ID: <50e39f698a7c0cc06d3bc060e6dbc539ea693241.1646905589.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We have a `fsync_component_or_die()` helper function which only syncs changes to disk in case the corresponding config is enabled by the user. This wrapper will always die on an error though, which makes it insufficient for new callsites we are about to add. Add a new `fsync_component()` wrapper which returns an error code instead of dying. Signed-off-by: Patrick Steinhardt --- cache.h | 13 ++++++++++--- write-or-die.c | 10 ++++++---- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/cache.h b/cache.h index f307e89516..63a95d1977 100644 --- a/cache.h +++ b/cache.h @@ -1745,12 +1745,19 @@ int copy_file(const char *dst, const char *src, int mode); int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); -void fsync_or_die(int fd, const char *); +int maybe_fsync(int fd); + +static inline int fsync_component(enum fsync_component component, int fd) +{ + if (fsync_components & component) + return maybe_fsync(fd); + return 0; +} static inline void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) { - if (fsync_components & component) - fsync_or_die(fd, msg); + if (fsync_component(component, fd) < 0) + die_errno("fsync error on '%s'", msg); } ssize_t read_in_full(int fd, void *buf, size_t count); diff --git a/write-or-die.c b/write-or-die.c index 9faa5f9f56..4a5455ce46 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -56,19 +56,21 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) } } -void fsync_or_die(int fd, const char *msg) +int maybe_fsync(int fd) { if (use_fsync < 0) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) - return; + return 0; if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) - return; + return 0; if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) - die_errno("fsync error on '%s'", msg); + return -1; + + return 0; } void write_or_die(int fd, const void *buf, size_t count) From patchwork Thu Mar 10 09:53:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12776112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ED0BC433EF for ; Thu, 10 Mar 2022 09:54:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241124AbiCJJy4 (ORCPT ); Thu, 10 Mar 2022 04:54:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241151AbiCJJyq (ORCPT ); Thu, 10 Mar 2022 04:54:46 -0500 Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EAA598F59 for ; Thu, 10 Mar 2022 01:53:25 -0800 (PST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 903FC5C01E6; Thu, 10 Mar 2022 04:53:24 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Thu, 10 Mar 2022 04:53:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=FFDMiVkOU7xGuOqudTofeRfhNKddc6mIXPHPCc XA4uM=; b=tAIZln4kA5jnzNLvUJdCFGGU9vefXkJaEK9LvLiWg3kS81rY7kQwTi CUEHEDN42MtUUWFkYqk/YhZEoqTp5d9XBze9kpHSjs7ySPe9diOdjvFY46hiQtFb XBC4zSwD8+DxjIr2wIrOc9TwojDLfPsQ4pJqcWMZbcEG704d2iKQjB/hd5q/Vy2j +Gflm/n06eQk5zbKa0aPBbTxPz54g2NeG1m2kkW7rsP6CxjhRE+Szq/DRhiY+4SY TfGxeq4274hNX9EvXkTT4TnEX2HwpFvfzLnZJAE1ytTjK2YUJPDeZpKj6sIdcW5D t2zguKNXpYGT0g9aZPGSyTlRT05j350Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=FFDMiVkOU7xGuOqud TofeRfhNKddc6mIXPHPCcXA4uM=; b=MOABwtW3BTBS4W+V22HyFb6Mn2bUH84yM NsG6KLvWRun4+/PhqxmnKkOdopBP0IlN41XPu9BmlM+jQX15YG9OrY0UnCj91Nw0 PPoaB5BKAo0KSb3ZVTLhzMIdsGgRZkKiFMKcg7g/JVmIaNyWPFWzoSSTt8GI0fdp 90e5L9QTyaZBnuMpfpiljIXDCi1wkkEdseVeDPiiYpeY+QGoia3QKRKHBV/YoDnR /f51xryXefIpzj5viOZik1c7aqWsP9GmrMiYzlYr2W/0REAgppabjq+0QdBuxIQI khUINEW7yFywexG38U4iwgVTFG1JVDRZ43Lg54mq98JiGRRm+bXQQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddruddvtddgtdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeetteegffefgfffjeehjeehjedvlefhkedtffegvdekfeevkeettdefudduffegieen ucffohhmrghinhepthhhuhhnkhdrohhrghdpkhgvrhhnvghlrdhorhhgnecuvehluhhsth gvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Mar 2022 04:53:22 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 8abfc7a4 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Mar 2022 09:53:22 +0000 (UTC) Date: Thu, 10 Mar 2022 10:53:21 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Junio C Hamano Subject: [PATCH 7/8] core.fsync: new option to harden loose references Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing loose references to disk we first create a lockfile, write the updated value of the reference into that lockfile, and on commit we rename the file into place. According to filesystem developers, this behaviour is broken because applications should always sync data to disk before doing the final rename to ensure data consistency [1][2][3]. If applications fail to do this correctly, a hard crash of the machine can easily result in corrupted on-disk data. This kind of corruption can in fact be easily observed with Git when the machine hard-crashes shortly after writing loose references to disk. On machines with ext4, this will likely lead to the "empty files" problem: the file has been renamed, but its data has not been synced to disk. The result is that the references is corrupt, and in the worst case this can lead to data loss. Implement a new option to harden loose references so that users and admins can avoid this scenario by syncing locked loose references to disk before we rename them into place. [1]: https://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/ [2]: https://btrfs.wiki.kernel.org/index.php/FAQ (What are the crash guarantees of overwrite-by-rename) [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/ext4.rst (see auto_da_alloc) Signed-off-by: Patrick Steinhardt --- Documentation/config/core.txt | 2 ++ cache.h | 6 +++++- config.c | 2 ++ refs/files-backend.c | 29 +++++++++++++++++++++++++++++ 4 files changed, 38 insertions(+), 1 deletion(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 973805e8a9..b67d3c340e 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -564,8 +564,10 @@ core.fsync:: * `pack-metadata` hardens packfile bitmaps and indexes. * `commit-graph` hardens the commit graph file. * `index` hardens the index when it is modified. +* `loose-ref` hardens references modified in the repo in loose-ref form. * `objects` is an aggregate option that is equivalent to `loose-object,pack`. +* `refs` is an aggregate option that is equivalent to `loose-ref`. * `derived-metadata` is an aggregate option that is equivalent to `pack-metadata,commit-graph`. * `default` is an aggregate option that is equivalent to diff --git a/cache.h b/cache.h index 63a95d1977..b56a56f539 100644 --- a/cache.h +++ b/cache.h @@ -1005,11 +1005,14 @@ enum fsync_component { FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, FSYNC_COMPONENT_INDEX = 1 << 4, + FSYNC_COMPONENT_LOOSE_REF = 1 << 5, }; #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK) +#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF) + #define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH) @@ -1026,7 +1029,8 @@ enum fsync_component { FSYNC_COMPONENT_PACK | \ FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH | \ - FSYNC_COMPONENT_INDEX) + FSYNC_COMPONENT_INDEX | \ + FSYNC_COMPONENT_LOOSE_REF) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index f03f27c3de..b5d3e6e404 100644 --- a/config.c +++ b/config.c @@ -1332,7 +1332,9 @@ static const struct fsync_component_entry { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "loose-ref", FSYNC_COMPONENT_LOOSE_REF }, { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "refs", FSYNC_COMPONENTS_REFS }, { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, { "default", FSYNC_COMPONENTS_DEFAULT }, { "committed", FSYNC_COMPONENTS_COMMITTED }, diff --git a/refs/files-backend.c b/refs/files-backend.c index f59589d6cc..279316de45 100644 --- a/refs/files-backend.c +++ b/refs/files-backend.c @@ -1392,6 +1392,15 @@ static int refs_rename_ref_available(struct ref_store *refs, return ok; } +static int files_sync_loose_ref(struct ref_lock *lock, struct strbuf *err) +{ + int ret = fsync_component(FSYNC_COMPONENT_LOOSE_REF, get_lock_file_fd(&lock->lk)); + if (ret) + strbuf_addf(err, "could not sync loose ref '%s': %s", lock->ref_name, + strerror(errno)); + return ret; +} + static int files_copy_or_rename_ref(struct ref_store *ref_store, const char *oldrefname, const char *newrefname, const char *logmsg, int copy) @@ -1504,6 +1513,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, oidcpy(&lock->old_oid, &orig_oid); if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || + files_sync_loose_ref(lock, &err) || commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) { error("unable to write current sha1 into %s: %s", newrefname, err.buf); strbuf_release(&err); @@ -1524,6 +1534,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store, flag = log_all_ref_updates; log_all_ref_updates = LOG_REFS_NONE; if (write_ref_to_lockfile(lock, &orig_oid, 0, &err) || + files_sync_loose_ref(lock, &err) || commit_ref_update(refs, lock, &orig_oid, NULL, &err)) { error("unable to write current sha1 into %s: %s", oldrefname, err.buf); strbuf_release(&err); @@ -2819,6 +2830,24 @@ static int files_transaction_prepare(struct ref_store *ref_store, } } + /* + * Sync all lockfiles to disk to ensure data consistency. We do this in + * a separate step such that we can sync all modified refs in a single + * step, which may be more efficient on some filesystems. + */ + for (i = 0; i < transaction->nr; i++) { + struct ref_update *update = transaction->updates[i]; + struct ref_lock *lock = update->backend_data; + + if (!(update->flags & REF_NEEDS_COMMIT)) + continue; + + if (files_sync_loose_ref(lock, err)) { + ret = TRANSACTION_GENERIC_ERROR; + goto cleanup; + } + } + cleanup: free(head_ref); string_list_clear(&affected_refnames, 0); From patchwork Thu Mar 10 09:53:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 12776113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 329C7C433F5 for ; Thu, 10 Mar 2022 09:54:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238644AbiCJJzE (ORCPT ); Thu, 10 Mar 2022 04:55:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238655AbiCJJys (ORCPT ); Thu, 10 Mar 2022 04:54:48 -0500 Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92978A76E5 for ; Thu, 10 Mar 2022 01:53:29 -0800 (PST) Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 583185C02E9; Thu, 10 Mar 2022 04:53:28 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Thu, 10 Mar 2022 04:53:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; bh=q18b+5VSi1MV8KTGfDFCwWYFCf+Tbx8lhi4pLq neqXY=; b=YmNjlIjRTsPx0BfZjZHL3lkjd7GizQup6WcqlWrUTjpfYqSpAzYvNW Xhrsxr2y4wqwls+YVH5jVP6qNw/OjGQ5kd99LRYUVPUtYmD4XEbDm0BW2iYKpm4Y sWMFEqztaINI+jtnt8yNQtW48WOMnP9KsnL0wU0A8iqRSHIq7pBtcfyK27Z8j+2u MsoBI4bGsFim/By+CGIn9lm62J/RadBwDcd7W0P+uEOuYrtV/FaOJzbCRzl0MgK1 7akmsh5JbEqf2Z5YRK2NWoTDE21Rvhnrov6AKTxg57rx6EYeWlhtBNiqtvwzV8gs nGUdE4haWXCxJzNyqtpU6Ty+/KDLwF1A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=q18b+5VSi1MV8KTGf DFCwWYFCf+Tbx8lhi4pLqneqXY=; b=XVEEdSmYO4WVUuxjaSGa0KDijPzk04LJg pBoH7wD6pc1B5wCtnvMZo/5FZbjyb7pQ4fIZPBywCDNxV9Mso3optR9w7P3WvfNo tU/tsfzQAjJ87rpEANSSLL+FmAq98ohS214qFoZXMmuumHrwSDJsLw7y3kIgf/jJ WhIOweh2RbdLMvzg8zXQO69yheKimuZBrqA0T087NSDCBpPZSX4WDjV3XH5iB39M 2v2MPM+DL8nSqjCx/Dw0cWK3m5fqAf90lp5Hf16v/7yxvfACQYQNmNH1LwE33ZOX AZaW22ozl0A40Hv0cBA/xbav4LbG8w99tYMHfpHCJm6yQ+P6y09mg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddruddvtddgtdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeehgefhtdefueffheekgfffudelffejtdfhvdejkedthfehvdelgfetgfdvtedthfen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 10 Mar 2022 04:53:26 -0500 (EST) Received: from localhost (ncase [10.192.0.11]) by vm-mail.pks.im (OpenSMTPD) with ESMTPSA id 50703f50 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 10 Mar 2022 09:53:26 +0000 (UTC) Date: Thu, 10 Mar 2022 10:53:25 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, sandals@crustytoothpaste.net, "Neeraj K. Singh" , Junio C Hamano Subject: [PATCH 8/8] core.fsync: new option to harden packed references Message-ID: <3b81d8f5aeffb73a32b0bff0da947f023a3df517.1646905589.git.ps@pks.im> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Similar to the preceding commit, this commit adds a new option to harden packed references so that users and admins can avoid data loss when we commit a new packed-refs file. Signed-off-by: Patrick Steinhardt --- Documentation/config/core.txt | 3 ++- cache.h | 7 +++++-- config.c | 1 + refs/packed-backend.c | 3 ++- 4 files changed, 10 insertions(+), 4 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index b67d3c340e..3fd466f955 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -565,9 +565,10 @@ core.fsync:: * `commit-graph` hardens the commit graph file. * `index` hardens the index when it is modified. * `loose-ref` hardens references modified in the repo in loose-ref form. +* `packed-refs` hardens references modified in the repo in packed-refs form. * `objects` is an aggregate option that is equivalent to `loose-object,pack`. -* `refs` is an aggregate option that is equivalent to `loose-ref`. +* `refs` is an aggregate option that is equivalent to `loose-ref,packed-refs`. * `derived-metadata` is an aggregate option that is equivalent to `pack-metadata,commit-graph`. * `default` is an aggregate option that is equivalent to diff --git a/cache.h b/cache.h index b56a56f539..9b7c282fa5 100644 --- a/cache.h +++ b/cache.h @@ -1006,12 +1006,14 @@ enum fsync_component { FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, FSYNC_COMPONENT_INDEX = 1 << 4, FSYNC_COMPONENT_LOOSE_REF = 1 << 5, + FSYNC_COMPONENT_PACKED_REFS = 1 << 6, }; #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ FSYNC_COMPONENT_PACK) -#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF) +#define FSYNC_COMPONENTS_REFS (FSYNC_COMPONENT_LOOSE_REF | \ + FSYNC_COMPONENT_PACKED_REFS) #define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH) @@ -1030,7 +1032,8 @@ enum fsync_component { FSYNC_COMPONENT_PACK_METADATA | \ FSYNC_COMPONENT_COMMIT_GRAPH | \ FSYNC_COMPONENT_INDEX | \ - FSYNC_COMPONENT_LOOSE_REF) + FSYNC_COMPONENT_LOOSE_REF | \ + FSYNC_COMPONENT_PACKED_REFS) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index b5d3e6e404..b4a2ee3a8c 100644 --- a/config.c +++ b/config.c @@ -1333,6 +1333,7 @@ static const struct fsync_component_entry { { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, { "loose-ref", FSYNC_COMPONENT_LOOSE_REF }, + { "packed-refs", FSYNC_COMPONENT_PACKED_REFS }, { "objects", FSYNC_COMPONENTS_OBJECTS }, { "refs", FSYNC_COMPONENTS_REFS }, { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, diff --git a/refs/packed-backend.c b/refs/packed-backend.c index 27dd8c3922..32d6635969 100644 --- a/refs/packed-backend.c +++ b/refs/packed-backend.c @@ -1262,7 +1262,8 @@ static int write_with_updates(struct packed_ref_store *refs, goto error; } - if (close_tempfile_gently(refs->tempfile)) { + if (fsync_component(FSYNC_COMPONENT_PACKED_REFS, get_tempfile_fd(refs->tempfile)) || + close_tempfile_gently(refs->tempfile)) { strbuf_addf(err, "error closing file %s: %s", get_tempfile_path(refs->tempfile), strerror(errno));