From patchwork Thu Mar 10 22:43:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12776983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1513C433EF for ; Thu, 10 Mar 2022 22:43:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343926AbiCJWod (ORCPT ); Thu, 10 Mar 2022 17:44:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239006AbiCJWoc (ORCPT ); Thu, 10 Mar 2022 17:44:32 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA380240B5 for ; Thu, 10 Mar 2022 14:43:29 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id n33-20020a05600c3ba100b003832caf7f3aso5184530wms.0 for ; Thu, 10 Mar 2022 14:43:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SYP+Yn4mhTuFFgy93xPQfv8JKnrUaPWyCv8xZy29qmc=; b=njYVStTW40gs9X36MxwmLZW7k8mCH87PS791rfYi6RVadHnKGgiLIjRE1Bcilqqfmc yAJOzLFQIKsneI4qtS0LxgAdAPIRFoiEx+021haNGpLUUukfhaVaQtSgdIeH+koR8Hfv Qc187jpRiQBnsa07Y+ca+o6ISqYLncoNM2NJuvQIdZU5ih7vVvW4AlagI6BAjSll9hMc cAWTjDH1jVhxRSxetZ1XepweyYVHrRAuCSQblBxMjuXCENKot7PJYzQtEKSzknhKZgdh l1J5Es48LmMl9bno1XiRSz2vwFSVcdGncya9+IJIh9eqM93ysLZVw+7kOZ6p4Ai8A6a1 MiZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SYP+Yn4mhTuFFgy93xPQfv8JKnrUaPWyCv8xZy29qmc=; b=L9EtMesdev5UG3hytfQwwWltw3ISGV+KJg9mJ8G09Jy+adbg9vZatya3D5jT60fTcP x/pmswYSGYcvIv35L7PrKSUhpe5DvL1cHZDEauNJl/CuHJv8r38DjD85NTLFdTVQaLJo 7gP4/iYfkXHVFITwyDj3xSetoTTVe0jS6DQbjkTtLJznR/lUlcbo17DJP0nGZLsIm5rY PEM/b1F6P9EUHMHdad2zZB0aUtqr+k7EJxaHL8w3j0+mbtAPNfk5+u5tHT1AjadNP42T tgszHlsAyOJ3Lf/FpWFG1ywAnJz4thUtVjdCDAx5v+855uJq5NbShUJ9STIenkwVcwsk Llgg== X-Gm-Message-State: AOAM532iBMcjpkt1q58D0OH4rz+Pp0yHGOQiSN1Dlp8Ew0UfNdpq04jS FllOODrhvRinm5abNMp7FaYR2ehc0TE= X-Google-Smtp-Source: ABdhPJxGu64CL9vTQsaiEcCWLqHQQAzcfFcX3wEbKtZj8VFoZFsa9KarihVw3b7NBSr4kktbUCqTcQ== X-Received: by 2002:a05:600c:4f09:b0:389:cf43:da62 with SMTP id l9-20020a05600c4f0900b00389cf43da62mr9008496wmq.204.1646952208051; Thu, 10 Mar 2022 14:43:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j42-20020a05600c1c2a00b00389d2ca24c9sm5542361wms.30.2022.03.10.14.43.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 14:43:27 -0800 (PST) Message-Id: <825079b6aa14c3975254a336e0e313b963bc9ab4.1646952205.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 10 Mar 2022 22:43:19 +0000 Subject: [PATCH v6 1/6] wrapper: make inclusion of Windows csprng header tightly scoped Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, Johannes Schindelin , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh Including NTSecAPI.h in git-compat-util.h causes build errors in any other file that includes winternl.h. NTSecAPI.h was included in order to get access to the RtlGenRandom cryptographically secure PRNG. This change scopes the inclusion of ntsecapi.h to wrapper.c, which is the only place that it's actually needed. The build breakage is due to the definition of UNICODE_STRING in NtSecApi.h: #ifndef _NTDEF_ typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING; typedef LSA_STRING STRING, *PSTRING ; #endif LsaLookup.h: typedef struct _LSA_UNICODE_STRING { USHORT Length; USHORT MaximumLength; #ifdef MIDL_PASS [size_is(MaximumLength/2), length_is(Length/2)] #endif // MIDL_PASS PWSTR Buffer; } LSA_UNICODE_STRING, *PLSA_UNICODE_STRING; winternl.h also defines UNICODE_STRING: typedef struct _UNICODE_STRING { USHORT Length; USHORT MaximumLength; PWSTR Buffer; } UNICODE_STRING; typedef UNICODE_STRING *PUNICODE_STRING; Both definitions have equivalent layouts. Apparently these internal Windows headers aren't designed to be included together. This is an oversight in the headers and does not represent an incompatibility between the APIs. Signed-off-by: Neeraj Singh --- compat/winansi.c | 5 ----- git-compat-util.h | 6 ------ wrapper.c | 7 +++++++ 3 files changed, 7 insertions(+), 11 deletions(-) diff --git a/compat/winansi.c b/compat/winansi.c index 936a80a5f00..3abe8dd5a27 100644 --- a/compat/winansi.c +++ b/compat/winansi.c @@ -4,11 +4,6 @@ #undef NOGDI -/* - * Including the appropriate header file for RtlGenRandom causes MSVC to see a - * redefinition of types in an incompatible way when including headers below. - */ -#undef HAVE_RTLGENRANDOM #include "../git-compat-util.h" #include #include diff --git a/git-compat-util.h b/git-compat-util.h index 876907b9df4..d210cff058c 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -197,12 +197,6 @@ #endif #include #define GIT_WINDOWS_NATIVE -#ifdef HAVE_RTLGENRANDOM -/* This is required to get access to RtlGenRandom. */ -#define SystemFunction036 NTAPI SystemFunction036 -#include -#undef SystemFunction036 -#endif #endif #include diff --git a/wrapper.c b/wrapper.c index 3258cdb171f..1108e4840a4 100644 --- a/wrapper.c +++ b/wrapper.c @@ -4,6 +4,13 @@ #include "cache.h" #include "config.h" +#ifdef HAVE_RTLGENRANDOM +/* This is required to get access to RtlGenRandom. */ +#define SystemFunction036 NTAPI SystemFunction036 +#include +#undef SystemFunction036 +#endif + static int memory_limit_check(size_t size, int gentle) { static size_t limit = 0; From patchwork Thu Mar 10 22:43:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12776987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82852C433F5 for ; Thu, 10 Mar 2022 22:43:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344440AbiCJWoq (ORCPT ); Thu, 10 Mar 2022 17:44:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242589AbiCJWog (ORCPT ); Thu, 10 Mar 2022 17:44:36 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEF45C683E for ; Thu, 10 Mar 2022 14:43:30 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id v2-20020a7bcb42000000b0037b9d960079so6457830wmj.0 for ; Thu, 10 Mar 2022 14:43:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7qT6iN3HPqdeOMExLnyPYtO1Aq2eT+8v+9G7h6ywli8=; b=IxW7rEh4yROH2b4sVBkDMHICkeupOAOCrrWMsazkrJTBoI9SPCrgmeCRv4MH4GJqoU u0xn10FlFUWzlZHvWthECGkSoFVJ6Tez0aFIo9yrXJsoru8K8ntdEOK0bnTh++bSTthk lv+Q2oCn+sjcNekrELh8V7v7qxt+rliaWOfdir/pAzxhM0u5f0tqw+JK+GJAA2k0a8Oc /sunifTp6VjQ8Ex/QNoxRAgTVGO4gj+cRr+Q3SCAcRtuHFEmHuy0/bhE2ljq8GY7qsg/ eYp0Vzy0pDqS3IyeFZAwSIMjIT33FdV58X+9ObEVW7P96eS2l9rKQg6mPz12QmPb9hU+ 8weQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7qT6iN3HPqdeOMExLnyPYtO1Aq2eT+8v+9G7h6ywli8=; b=Kq4TwSWm3K8OqElVGaMubHL/WKs8hPpCXohepitQCPrn7orHTD/vcf3vK2Eg1XclgQ U+/Akswtcarb86WGdzsddILlrP51Of4FRK8aiD2hlw30eok1e21GP6YpX21utwkLiknr d3yQuGf3dWDauIbhCJun73okMklUnX+8jiKfa+cjnN3/t5RIzkllYHJ3BMIp9o+ko0Hh PDeOID0iHdS8hgl/9yodx71aJ0Ftu+k7JOOgkPrceK6csQ3FbeRIRp/nAMFTMTYPO9Ez oec2ZyLy9VhnlmNjtHi8olXAosboIT/RIX15dx5FBWgpz51s4440egPO/AvT9FHGZZoo VYKQ== X-Gm-Message-State: AOAM5302+zIotOZLJFz92WF/7QD+p8B7mU64gbueU4qHFIgyj/o4RXDo y04FFLYdbYUoQ7QG0m1Db5X4YJm7xcY= X-Google-Smtp-Source: ABdhPJwaiCIz7gal9GjcQcHC1uB4PZCgVwim2P0MbB5MOvouLrQc2iUgDAYpnZRTatgKtengBfBQMg== X-Received: by 2002:a05:600c:4fc7:b0:389:ec8c:b544 with SMTP id o7-20020a05600c4fc700b00389ec8cb544mr353097wmq.202.1646952209206; Thu, 10 Mar 2022 14:43:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a8-20020a05600c068800b00389bdc8c8c2sm5293681wmn.12.2022.03.10.14.43.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 14:43:28 -0800 (PST) Message-Id: In-Reply-To: References: Date: Thu, 10 Mar 2022 22:43:20 +0000 Subject: [PATCH v6 2/6] core.fsyncmethod: add writeout-only mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, Johannes Schindelin , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 9 ++++ Makefile | 6 +++ cache.h | 7 ++++ compat/mingw.h | 3 ++ compat/win32/flush.c | 28 +++++++++++++ config.c | 12 ++++++ config.mak.uname | 3 ++ configure.ac | 8 ++++ contrib/buildsystems/CMakeLists.txt | 16 ++++++-- environment.c | 1 + git-compat-util.h | 24 +++++++++++ wrapper.c | 64 +++++++++++++++++++++++++++++ write-or-die.c | 11 +++-- 13 files changed, 184 insertions(+), 8 deletions(-) create mode 100644 compat/win32/flush.c diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..dbb134f7136 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,15 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsyncMethod:: + A value indicating the strategy Git will use to harden repository data + using fsync and related primitives. ++ +* `fsync` uses the fsync() system call or platform equivalents. +* `writeout-only` issues pagecache writeback requests, but depending on the + filesystem and storage hardware, data added to the repository may not be + durable in the event of a system crash. This is the default mode on macOS. + core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + diff --git a/Makefile b/Makefile index 6f0b4b775fe..17fd9b023a4 100644 --- a/Makefile +++ b/Makefile @@ -411,6 +411,8 @@ all:: # # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC. # +# Define HAVE_SYNC_FILE_RANGE if your platform has sync_file_range. +# # Define NEEDS_LIBRT if your platform requires linking with librt (glibc version # before 2.17) for clock_gettime and CLOCK_MONOTONIC. # @@ -1897,6 +1899,10 @@ ifdef HAVE_CLOCK_MONOTONIC BASIC_CFLAGS += -DHAVE_CLOCK_MONOTONIC endif +ifdef HAVE_SYNC_FILE_RANGE + BASIC_CFLAGS += -DHAVE_SYNC_FILE_RANGE +endif + ifdef NEEDS_LIBRT EXTLIBS += -lrt endif diff --git a/cache.h b/cache.h index 04d4d2db25c..82f0194a3dd 100644 --- a/cache.h +++ b/cache.h @@ -995,6 +995,13 @@ extern char *git_replace_ref_base; extern int fsync_object_files; extern int use_fsync; + +enum fsync_method { + FSYNC_METHOD_FSYNC, + FSYNC_METHOD_WRITEOUT_ONLY +}; + +extern enum fsync_method fsync_method; extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; diff --git a/compat/mingw.h b/compat/mingw.h index c9a52ad64a6..6074a3d3ced 100644 --- a/compat/mingw.h +++ b/compat/mingw.h @@ -329,6 +329,9 @@ int mingw_getpagesize(void); #define getpagesize mingw_getpagesize #endif +int win32_fsync_no_flush(int fd); +#define fsync_no_flush win32_fsync_no_flush + struct rlimit { unsigned int rlim_cur; }; diff --git a/compat/win32/flush.c b/compat/win32/flush.c new file mode 100644 index 00000000000..291f90ea940 --- /dev/null +++ b/compat/win32/flush.c @@ -0,0 +1,28 @@ +#include "git-compat-util.h" +#include +#include "lazyload.h" + +int win32_fsync_no_flush(int fd) +{ + IO_STATUS_BLOCK io_status; + +#define FLUSH_FLAGS_FILE_DATA_ONLY 1 + + DECLARE_PROC_ADDR(ntdll.dll, NTSTATUS, NTAPI, NtFlushBuffersFileEx, + HANDLE FileHandle, ULONG Flags, PVOID Parameters, ULONG ParameterSize, + PIO_STATUS_BLOCK IoStatusBlock); + + if (!INIT_PROC_ADDR(NtFlushBuffersFileEx)) { + errno = ENOSYS; + return -1; + } + + memset(&io_status, 0, sizeof(io_status)); + if (NtFlushBuffersFileEx((HANDLE)_get_osfhandle(fd), FLUSH_FLAGS_FILE_DATA_ONLY, + NULL, 0, &io_status)) { + errno = EINVAL; + return -1; + } + + return 0; +} diff --git a/config.c b/config.c index 383b1a4885b..f3ff80b01c9 100644 --- a/config.c +++ b/config.c @@ -1600,6 +1600,18 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsyncmethod")) { + if (!value) + return config_error_nonbool(var); + if (!strcmp(value, "fsync")) + fsync_method = FSYNC_METHOD_FSYNC; + else if (!strcmp(value, "writeout-only")) + fsync_method = FSYNC_METHOD_WRITEOUT_ONLY; + else + warning(_("ignoring unknown core.fsyncMethod value '%s'"), value); + + } + if (!strcmp(var, "core.fsyncobjectfiles")) { fsync_object_files = git_config_bool(var, value); return 0; diff --git a/config.mak.uname b/config.mak.uname index 4352ea39e9b..404fff5dd04 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -57,6 +57,7 @@ ifeq ($(uname_S),Linux) HAVE_CLOCK_MONOTONIC = YesPlease # -lrt is needed for clock_gettime on glibc <= 2.16 NEEDS_LIBRT = YesPlease + HAVE_SYNC_FILE_RANGE = YesPlease HAVE_GETDELIM = YesPlease FREAD_READS_DIRECTORIES = UnfortunatelyYes BASIC_CFLAGS += -DHAVE_SYSINFO @@ -463,6 +464,7 @@ endif CFLAGS = BASIC_CFLAGS = -nologo -I. -Icompat/vcbuild/include -DWIN32 -D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE COMPAT_OBJS = compat/msvc.o compat/winansi.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/trace2_win32_process_info.o \ @@ -640,6 +642,7 @@ ifeq ($(uname_S),MINGW) COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\" COMPAT_OBJS += compat/mingw.o compat/winansi.o \ compat/win32/trace2_win32_process_info.o \ + compat/win32/flush.o \ compat/win32/path-utils.o \ compat/win32/pthread.o compat/win32/syslog.o \ compat/win32/dirent.o diff --git a/configure.ac b/configure.ac index 5ee25ec95c8..6bd6bef1c44 100644 --- a/configure.ac +++ b/configure.ac @@ -1082,6 +1082,14 @@ AC_COMPILE_IFELSE([CLOCK_MONOTONIC_SRC], [AC_MSG_RESULT([no]) HAVE_CLOCK_MONOTONIC=]) GIT_CONF_SUBST([HAVE_CLOCK_MONOTONIC]) + +# +# Define HAVE_SYNC_FILE_RANGE=YesPlease if sync_file_range is available. +GIT_CHECK_FUNC(sync_file_range, + [HAVE_SYNC_FILE_RANGE=YesPlease], + [HAVE_SYNC_FILE_RANGE]) +GIT_CONF_SUBST([HAVE_SYNC_FILE_RANGE]) + # # Define NO_SETITIMER if you don't have setitimer. GIT_CHECK_FUNC(setitimer, diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index e44232f85d3..3a9e6241660 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -261,10 +261,18 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Windows") NOGDI OBJECT_CREATION_MODE=1 __USE_MINGW_ANSI_STDIO=0 USE_NED_ALLOCATOR OVERRIDE_STRDUP MMAP_PREVENTS_DELETE USE_WIN32_MMAP UNICODE _UNICODE HAVE_WPGMPTR ENSURE_MSYSTEM_IS_SET HAVE_RTLGENRANDOM) - list(APPEND compat_SOURCES compat/mingw.c compat/winansi.c compat/win32/path-utils.c - compat/win32/pthread.c compat/win32mmap.c compat/win32/syslog.c - compat/win32/trace2_win32_process_info.c compat/win32/dirent.c - compat/nedmalloc/nedmalloc.c compat/strdup.c) + list(APPEND compat_SOURCES + compat/mingw.c + compat/winansi.c + compat/win32/flush.c + compat/win32/path-utils.c + compat/win32/pthread.c + compat/win32mmap.c + compat/win32/syslog.c + compat/win32/trace2_win32_process_info.c + compat/win32/dirent.c + compat/nedmalloc/nedmalloc.c + compat/strdup.c) set(NO_UNIX_SOCKETS 1) elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") diff --git a/environment.c b/environment.c index fd0501e77a5..3e3620d759f 100644 --- a/environment.c +++ b/environment.c @@ -44,6 +44,7 @@ int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; int fsync_object_files; int use_fsync = -1; +enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/git-compat-util.h b/git-compat-util.h index d210cff058c..00356476a9d 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -1271,6 +1271,30 @@ __attribute__((format (printf, 1, 2))) NORETURN void BUG(const char *fmt, ...); #endif +#ifdef __APPLE__ +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_WRITEOUT_ONLY +#else +#define FSYNC_METHOD_DEFAULT FSYNC_METHOD_FSYNC +#endif + +enum fsync_action { + FSYNC_WRITEOUT_ONLY, + FSYNC_HARDWARE_FLUSH +}; + +/* + * Issues an fsync against the specified file according to the specified mode. + * + * FSYNC_WRITEOUT_ONLY attempts to use interfaces available on some operating + * systems to flush the OS cache without issuing a flush command to the storage + * controller. If those interfaces are unavailable, the function fails with + * ENOSYS. + * + * FSYNC_HARDWARE_FLUSH does an OS writeout and hardware flush to ensure that + * changes are durable. It is not expected to fail. + */ +int git_fsync(int fd, enum fsync_action action); + /* * Preserves errno, prints a message, but gives no warning for ENOENT. * Returns 0 on success, which includes trying to unlink an object that does diff --git a/wrapper.c b/wrapper.c index 1108e4840a4..354d784c034 100644 --- a/wrapper.c +++ b/wrapper.c @@ -546,6 +546,70 @@ int xmkstemp_mode(char *filename_template, int mode) return fd; } +/* + * Some platforms return EINTR from fsync. Since fsync is invoked in some + * cases by a wrapper that dies on failure, do not expose EINTR to callers. + */ +static int fsync_loop(int fd) +{ + int err; + + do { + err = fsync(fd); + } while (err < 0 && errno == EINTR); + return err; +} + +int git_fsync(int fd, enum fsync_action action) +{ + switch (action) { + case FSYNC_WRITEOUT_ONLY: + +#ifdef __APPLE__ + /* + * On macOS, fsync just causes filesystem cache writeback but + * does not flush hardware caches. + */ + return fsync_loop(fd); +#endif + +#ifdef HAVE_SYNC_FILE_RANGE + /* + * On linux 2.6.17 and above, sync_file_range is the way to + * issue a writeback without a hardware flush. An offset of + * 0 and size of 0 indicates writeout of the entire file and the + * wait flags ensure that all dirty data is written to the disk + * (potentially in a disk-side cache) before we continue. + */ + + return sync_file_range(fd, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE | + SYNC_FILE_RANGE_WRITE | + SYNC_FILE_RANGE_WAIT_AFTER); +#endif + +#ifdef fsync_no_flush + return fsync_no_flush(fd); +#endif + + errno = ENOSYS; + return -1; + + case FSYNC_HARDWARE_FLUSH: + /* + * On macOS, a special fcntl is required to really flush the + * caches within the storage controller. As of this writing, + * this is a very expensive operation on Apple SSDs. + */ +#ifdef __APPLE__ + return fcntl(fd, F_FULLFSYNC); +#else + return fsync_loop(fd); +#endif + default: + BUG("unexpected git_fsync(%d) call", action); + } +} + static int warn_if_unremovable(const char *op, const char *file, int rc) { int err; diff --git a/write-or-die.c b/write-or-die.c index a3d5784cec9..9faa5f9f563 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -62,10 +62,13 @@ void fsync_or_die(int fd, const char *msg) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) return; - while (fsync(fd) < 0) { - if (errno != EINTR) - die_errno("fsync error on '%s'", msg); - } + + if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && + git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) + return; + + if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) + die_errno("fsync error on '%s'", msg); } void write_or_die(int fd, const void *buf, size_t count) From patchwork Thu Mar 10 22:43:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12776984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14053C433F5 for ; Thu, 10 Mar 2022 22:43:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344418AbiCJWof (ORCPT ); Thu, 10 Mar 2022 17:44:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239006AbiCJWoe (ORCPT ); Thu, 10 Mar 2022 17:44:34 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B88D1C683F for ; Thu, 10 Mar 2022 14:43:31 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id bg31-20020a05600c3c9f00b00381590dbb33so4334489wmb.3 for ; Thu, 10 Mar 2022 14:43:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pOhylVgFye2yLzOGLZc3HYCA4zW3pMSMK67FqSTH73U=; b=Ts0d9X03S+gjFFNe8FvMJ2J8tq0zJ8XnbUsABIIgebrIDyLmJIGdlUUPtlaSCqZ5yn NIKscWxoaRq2Bi+/vwbTg4e/sdXgdiM2drAV6mcG3AqIsFWob59KkmZ4xNyIH9PO1Hbc E8Ma0nvScAHUMbMzkgVtRF8rOhy8PXSPf5ExhM5vpOB3lKXO12bYizIP2y9622EG712z njPAG2S40tshNpeuYLVIDBLs136fR4aHkpczDYCX98CFxMx2YCMpt1DV/syOwVAcIlY9 ehB0dz8qWcgUPoms1LxmbtUdp8Svznmq4fnaycBgrwtpeDgip0tsUVknJg3GndDk1nUU 1rUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pOhylVgFye2yLzOGLZc3HYCA4zW3pMSMK67FqSTH73U=; b=u2hIM/wSCP4ULlK37uj+KyK4hhKFqzME5GZ9zvWK1jSQJhcnw144HKbpTL2S6LdFsI +GNA570FX6Zsi9JmkDQKQSddCF2rT02aef7O+yJUUrAxB4uoRP1MakTP/5umfrVDLXz2 Nq8Z8i7TMHrRzq76dJHynruByAfewVrtue/jlS0QJ+wKhObUPTIvjUjxUF8JGGP7L+Om Ln5uj9ywddDz5F4Ci9XwfoIxVULUYVFAyTVlIC95kcJghyC0binQ7oor2NFcqimBbV8G 2egUq+HmFsCyk/xFTc0sLtXSysJCYgr921T6uKfhXGJ9gNQnAxH6NI39/R61nEb4tUsx qutQ== X-Gm-Message-State: AOAM5315debfnv4GOsEEhuRNWdO8o3bkS1Ne62NsGBE4ZPeMhuqvQjY7 4CfcemhfHiQdgqYU/oyt85Y3bEeKVT4= X-Google-Smtp-Source: ABdhPJx0S67EZ4SOC9cqrOqBpK7pvfh0rV17Ts38flZXDHwapwJudluUyZ3II99Y6T0OVD1NwtBIkg== X-Received: by 2002:a1c:f009:0:b0:387:6fea:8ebc with SMTP id a9-20020a1cf009000000b003876fea8ebcmr13301144wmb.84.1646952209966; Thu, 10 Mar 2022 14:43:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e2-20020adfe7c2000000b001f04d622e7fsm5039204wrn.39.2022.03.10.14.43.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 14:43:29 -0800 (PST) Message-Id: <64e2bdcdfd92ba643cee52605155c21d116710a7.1646952205.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 10 Mar 2022 22:43:21 +0000 Subject: [PATCH v6 3/6] core.fsync: introduce granular fsync control infrastructure Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, Johannes Schindelin , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit introduces the infrastructure for the core.fsync configuration knob. The repository components we want to sync are identified by flags so that we can turn on or off syncing for specific components. If core.fsyncObjectFiles is set and the core.fsync configuration also includes FSYNC_COMPONENT_LOOSE_OBJECT, we will fsync any loose objects. This picks the strictest data integrity behavior if core.fsync and core.fsyncObjectFiles are set to conflicting values. This change introduces the currently unused fsync_component helper, which will be used by a later patch that adds fsyncing to the refs backend. Actual configuration and documentation of the fsync components list are in other patches in the series to separate review of the underlying mechanism from the policy of how it's configured. Helped-by: Patrick Steinhardt Signed-off-by: Neeraj Singh --- builtin/fast-import.c | 2 +- builtin/index-pack.c | 4 ++-- builtin/pack-objects.c | 24 +++++++++++++++++------- bulk-checkin.c | 5 +++-- cache.h | 23 +++++++++++++++++++++++ commit-graph.c | 3 ++- csum-file.c | 5 +++-- csum-file.h | 3 ++- environment.c | 1 + midx.c | 3 ++- object-file.c | 13 +++++++++---- pack-bitmap-write.c | 3 ++- pack-write.c | 13 +++++++------ read-cache.c | 2 +- write-or-die.c | 26 ++++++++++++++++++++++---- 15 files changed, 97 insertions(+), 33 deletions(-) diff --git a/builtin/fast-import.c b/builtin/fast-import.c index b7105fcad9b..f2c036a8955 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -865,7 +865,7 @@ static void end_packfile(void) struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, 0); + finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(pack_data->pack_fd, pack_data->hash, pack_data->pack_name, object_count, cur_pack_oid.hash, pack_size); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index c45273de3b1..c5f12f14df5 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1290,7 +1290,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, 0); + finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); hashcpy(read_hash, pack_hash); fixup_pack_header_footer(output_fd, pack_hash, curr_pack, nr_objects, @@ -1512,7 +1512,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name, if (!from_stdin) { close(input_fd); } else { - fsync_or_die(output_fd, curr_pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, output_fd, curr_pack_name); err = close(output_fd); if (err) die_errno(_("error while closing pack file")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 178e611f09d..c14fee8e99f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1199,16 +1199,26 @@ static void write_pack_file(void) display_progress(progress_state, written); } - /* - * Did we write the wrong # entries in the header? - * If so, rewrite it like in fast-import - */ if (pack_to_stdout) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_CLOSE); + /* + * We never fsync when writing to stdout since we may + * not be writing to an actual pack file. For instance, + * the upload-pack code passes a pipe here. Calling + * fsync on a pipe results in unnecessary + * synchronization with the reader on some platforms. + */ + finalize_hashfile(f, hash, FSYNC_COMPONENT_NONE, + CSUM_HASH_IN_STREAM | CSUM_CLOSE); } else if (nr_written == nr_remaining) { - finalize_hashfile(f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(f, hash, 0); + /* + * If we wrote the wrong number of entries in the + * header, rewrite it like in fast-import. + */ + + int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); diff --git a/bulk-checkin.c b/bulk-checkin.c index 8785b2ac806..a2cf9dcbc8d 100644 --- a/bulk-checkin.c +++ b/bulk-checkin.c @@ -53,9 +53,10 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state) unlink(state->pack_tmp_name); goto clear_exit; } else if (state->nr_written == 1) { - finalize_hashfile(state->f, hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); } else { - int fd = finalize_hashfile(state->f, hash, 0); + int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(fd, hash, state->pack_tmp_name, state->nr_written, hash, state->offset); diff --git a/cache.h b/cache.h index 82f0194a3dd..7ac1959258d 100644 --- a/cache.h +++ b/cache.h @@ -993,6 +993,27 @@ void reset_shared_repository(void); extern int read_replace_refs; extern char *git_replace_ref_base; +/* + * These values are used to help identify parts of a repository to fsync. + * FSYNC_COMPONENT_NONE identifies data that will not be a persistent part of the + * repository and so shouldn't be fsynced. + */ +enum fsync_component { + FSYNC_COMPONENT_NONE, + FSYNC_COMPONENT_LOOSE_OBJECT = 1 << 0, + FSYNC_COMPONENT_PACK = 1 << 1, + FSYNC_COMPONENT_PACK_METADATA = 1 << 2, + FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, +}; + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +/* + * A bitmask indicating which components of the repo should be fsynced. + */ +extern enum fsync_component fsync_components; extern int fsync_object_files; extern int use_fsync; @@ -1707,6 +1728,8 @@ int copy_file_with_time(const char *dst, const char *src, int mode); void write_or_die(int fd, const void *buf, size_t count); void fsync_or_die(int fd, const char *); +int fsync_component(enum fsync_component component, int fd); +void fsync_component_or_die(enum fsync_component component, int fd, const char *msg); ssize_t read_in_full(int fd, void *buf, size_t count); ssize_t write_in_full(int fd, const void *buf, size_t count); diff --git a/commit-graph.c b/commit-graph.c index 265c010122e..64897f57d9f 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1942,7 +1942,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } close_commit_graph(ctx->r->objects); - finalize_hashfile(f, file_hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC); + finalize_hashfile(f, file_hash, FSYNC_COMPONENT_COMMIT_GRAPH, + CSUM_HASH_IN_STREAM | CSUM_FSYNC); free_chunkfile(cf); if (ctx->split) { diff --git a/csum-file.c b/csum-file.c index 26e8a6df44e..59ef3398ca2 100644 --- a/csum-file.c +++ b/csum-file.c @@ -58,7 +58,8 @@ static void free_hashfile(struct hashfile *f) free(f); } -int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int flags) +int finalize_hashfile(struct hashfile *f, unsigned char *result, + enum fsync_component component, unsigned int flags) { int fd; @@ -69,7 +70,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result, unsigned int fl if (flags & CSUM_HASH_IN_STREAM) flush(f, f->buffer, the_hash_algo->rawsz); if (flags & CSUM_FSYNC) - fsync_or_die(f->fd, f->name); + fsync_component_or_die(component, f->fd, f->name); if (flags & CSUM_CLOSE) { if (close(f->fd)) die_errno("%s: sha1 file error on close", f->name); diff --git a/csum-file.h b/csum-file.h index 291215b34eb..0d29f528fbc 100644 --- a/csum-file.h +++ b/csum-file.h @@ -1,6 +1,7 @@ #ifndef CSUM_FILE_H #define CSUM_FILE_H +#include "cache.h" #include "hash.h" struct progress; @@ -38,7 +39,7 @@ int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *); struct hashfile *hashfd(int fd, const char *name); struct hashfile *hashfd_check(const char *name); struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp); -int finalize_hashfile(struct hashfile *, unsigned char *, unsigned int); +int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int); void hashwrite(struct hashfile *, const void *, unsigned int); void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); diff --git a/environment.c b/environment.c index 3e3620d759f..36ca5fb2e77 100644 --- a/environment.c +++ b/environment.c @@ -45,6 +45,7 @@ int pack_compression_level = Z_DEFAULT_COMPRESSION; int fsync_object_files; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; +enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; diff --git a/midx.c b/midx.c index 865170bad05..107365d2114 100644 --- a/midx.c +++ b/midx.c @@ -1438,7 +1438,8 @@ static int write_midx_internal(const char *object_dir, write_midx_header(f, get_num_chunks(cf), ctx.nr - dropped_packs); write_chunkfile(cf, &ctx); - finalize_hashfile(f, midx_hash, CSUM_FSYNC | CSUM_HASH_IN_STREAM); + finalize_hashfile(f, midx_hash, FSYNC_COMPONENT_PACK_METADATA, + CSUM_FSYNC | CSUM_HASH_IN_STREAM); free_chunkfile(cf); if (flags & MIDX_WRITE_REV_INDEX && diff --git a/object-file.c b/object-file.c index 03bd6a3baf3..e3f0bf27ff1 100644 --- a/object-file.c +++ b/object-file.c @@ -1849,11 +1849,16 @@ int hash_object_file(const struct git_hash_algo *algo, const void *buf, /* Finalize a file on disk, and close it. */ static void close_loose_object(int fd) { - if (!the_repository->objects->odb->will_destroy) { - if (fsync_object_files) - fsync_or_die(fd, "loose object file"); - } + if (the_repository->objects->odb->will_destroy) + goto out; + if (fsync_object_files > 0) + fsync_or_die(fd, "loose object file"); + else + fsync_component_or_die(FSYNC_COMPONENT_LOOSE_OBJECT, fd, + "loose object file"); + +out: if (close(fd) != 0) die_errno(_("error when closing loose object file")); } diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index cab3eaa2acd..cf681547f2e 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -719,7 +719,8 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); if (adjust_shared_perm(tmp_file.buf)) die_errno("unable to make temporary bitmap file readable"); diff --git a/pack-write.c b/pack-write.c index a5846f3a346..51812cb1299 100644 --- a/pack-write.c +++ b/pack-write.c @@ -159,9 +159,9 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } hashwrite(f, sha1, the_hash_algo->rawsz); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((opts->flags & WRITE_IDX_VERIFY) - ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return index_name; } @@ -281,8 +281,9 @@ const char *write_rev_file_order(const char *rev_name, if (rev_name && adjust_shared_perm(rev_name) < 0) die(_("failed to make %s readable"), rev_name); - finalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_CLOSE | - ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, + CSUM_HASH_IN_STREAM | CSUM_CLOSE | + ((flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC)); return rev_name; } @@ -390,7 +391,7 @@ void fixup_pack_header_footer(int pack_fd, the_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx); the_hash_algo->final_fn(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz); - fsync_or_die(pack_fd, pack_name); + fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); } char *index_pack_lockfile(int ip_out, int *is_well_formed) diff --git a/read-cache.c b/read-cache.c index 79b9b99ebf7..df869691fd4 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3089,7 +3089,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, CSUM_HASH_IN_STREAM); + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; diff --git a/write-or-die.c b/write-or-die.c index 9faa5f9f563..c4fd91b5b43 100644 --- a/write-or-die.c +++ b/write-or-die.c @@ -56,21 +56,39 @@ void fprintf_or_die(FILE *f, const char *fmt, ...) } } -void fsync_or_die(int fd, const char *msg) +static int maybe_fsync(int fd) { if (use_fsync < 0) use_fsync = git_env_bool("GIT_TEST_FSYNC", 1); if (!use_fsync) - return; + return 0; if (fsync_method == FSYNC_METHOD_WRITEOUT_ONLY && git_fsync(fd, FSYNC_WRITEOUT_ONLY) >= 0) - return; + return 0; + + return git_fsync(fd, FSYNC_HARDWARE_FLUSH); +} - if (git_fsync(fd, FSYNC_HARDWARE_FLUSH) < 0) +void fsync_or_die(int fd, const char *msg) +{ + if (maybe_fsync(fd) < 0) die_errno("fsync error on '%s'", msg); } +int fsync_component(enum fsync_component component, int fd) +{ + if (fsync_components & component) + return maybe_fsync(fd); + return 0; +} + +void fsync_component_or_die(enum fsync_component component, int fd, const char *msg) +{ + if (fsync_components & component) + fsync_or_die(fd, msg); +} + void write_or_die(int fd, const void *buf, size_t count) { if (write_in_full(fd, buf, count) < 0) { From patchwork Thu Mar 10 22:43:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12776985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0523C433F5 for ; Thu, 10 Mar 2022 22:43:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344423AbiCJWol (ORCPT ); Thu, 10 Mar 2022 17:44:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344396AbiCJWoe (ORCPT ); Thu, 10 Mar 2022 17:44:34 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A6173A4 for ; Thu, 10 Mar 2022 14:43:32 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id l1-20020a05600c4f0100b00389645443d2so4348783wmq.2 for ; Thu, 10 Mar 2022 14:43:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5NNW0FbavEj2yXOFUjt6PGyxIlD+hPEoObXt6SjaOHs=; b=Ml08Yf04ONKK+I2psy53oOMFl3pz64zoQxQ8P+JZy0x8k13jN0PmpB4+6txW6zJOWQ lP8obkBwlsWc/RtiEt6RuS7dlETGNc8aVYlDviwi3uIQDu7k+30q8Apuy/iQjlKlfOcZ LCkgH2N0F6wz8jUD2FDNP7XcgDmb9U/ymULnjkndtzlKl97WvUL+GORUd/km1UYhZVIe 9KX5RfzkYQvKrXyKWAObm0p5mD2JyiZPVRjLxUfygTGHr9b0wE3l60SgbAS1BQ5Xzitn RNEJfeaU9Lf4uMZMaDGeTGBakyrEUf3v6iyozQYRpqNB2i3Co+VgVDfDIMK9Z1BEbK8Z Wdaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5NNW0FbavEj2yXOFUjt6PGyxIlD+hPEoObXt6SjaOHs=; b=5D9O/kHWaGOY5aPkZIc6qcvbMatBDg9PhweK7knMfilvknQ1/I7QYvQ7cB54mgRgtF 0wJkGZYCXlzV+ZYS6nPjQh6SOOJN5/sOEVzSN81GQw4St7tReXttdUsCMDkj7go+G11+ DH3cJ0yMMzztMy6UJyeUlHWBmS6Hfzg8YiDVTfbMmoN4taWdu2vELRW175iczcjpt/2Q q/tuPczPH8BxdsJw5G3sup5caTjcam8h8NbYQM0F9Q6g+VVw87ealjGB0bz1w+k8b4yF lDisrCev45sYXdedA36vBwJleEX9cJefoV9UUMJgHiUaUx9niur1wSw9otR5p2A4koEK 6vUw== X-Gm-Message-State: AOAM531XjOdN8kykbbCGTI+VyLG2K51lFa0QHCXalZFsQJdvUTlNNccq JfqFbz9gut+xxbV3DvZoMguLJ4Ez/sw= X-Google-Smtp-Source: ABdhPJxzb/oWOO3ccIkbStURt5SaI4ymaET5dS4ynjSw8IF9fm6BvBRYOGDg0iMNmx3LtXWynxpcJA== X-Received: by 2002:a05:600c:4a12:b0:389:9c7d:5917 with SMTP id c18-20020a05600c4a1200b003899c7d5917mr5273008wmp.0.1646952211014; Thu, 10 Mar 2022 14:43:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d8-20020a056000114800b001f045f3435asm7180570wrx.108.2022.03.10.14.43.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 14:43:30 -0800 (PST) Message-Id: <6adc8dc13852c219763a9830f848fbc8663f2fa9.1646952205.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 10 Mar 2022 22:43:22 +0000 Subject: [PATCH v6 4/6] core.fsync: add configuration parsing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, Johannes Schindelin , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This change introduces code to parse the core.fsync setting and configure the fsync_components variable. core.fsync is configured as a comma-separated list of component names to sync. Each time a core.fsync variable is encountered in the configuration heirarchy, we start off with a clean state with the platform default value. Passing 'none' resets the value to indicate nothing will be synced. We gather all negative and positive entries from the comma separated list and then compute the new value by removing all the negative entries and adding all of the positive entries. We issue a warning for components that are not recognized so that the configuration code is compatible with configs from future versions of Git with more repo components. Complete documentation for the new setting is included in a later patch in the series so that it can be reviewed once in final form. Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 9 +++-- config.c | 76 +++++++++++++++++++++++++++++++++++ environment.c | 2 +- 3 files changed, 82 insertions(+), 5 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index dbb134f7136..ab911d6e269 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -558,11 +558,12 @@ core.fsyncMethod:: core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + This setting is deprecated. Use core.fsync instead. + -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). +This setting affects data added to the Git repository in loose-object +form. When set to true, Git will issue an fsync or similar system call +to flush caches so that loose-objects remain consistent in the face +of a unclean system shutdown. core.preloadIndex:: Enable parallel index preload for operations like 'git diff' diff --git a/config.c b/config.c index f3ff80b01c9..94a4598b5fc 100644 --- a/config.c +++ b/config.c @@ -1323,6 +1323,73 @@ static int git_parse_maybe_bool_text(const char *value) return -1; } +static const struct fsync_component_name { + const char *name; + enum fsync_component component_bits; +} fsync_component_names[] = { + { "loose-object", FSYNC_COMPONENT_LOOSE_OBJECT }, + { "pack", FSYNC_COMPONENT_PACK }, + { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, + { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, +}; + +static enum fsync_component parse_fsync_components(const char *var, const char *string) +{ + enum fsync_component current = FSYNC_COMPONENTS_DEFAULT; + enum fsync_component positive = 0, negative = 0; + + while (string) { + int i; + size_t len; + const char *ep; + int negated = 0; + int found = 0; + + string = string + strspn(string, ", \t\n\r"); + ep = strchrnul(string, ','); + len = ep - string; + if (!strcmp(string, "none")) { + current = FSYNC_COMPONENT_NONE; + goto next_name; + } + + if (*string == '-') { + negated = 1; + string++; + len--; + if (!len) + warning(_("invalid value for variable %s"), var); + } + + if (!len) + break; + + for (i = 0; i < ARRAY_SIZE(fsync_component_names); ++i) { + const struct fsync_component_name *n = &fsync_component_names[i]; + + if (strncmp(n->name, string, len)) + continue; + + found = 1; + if (negated) + negative |= n->component_bits; + else + positive |= n->component_bits; + } + + if (!found) { + char *component = xstrndup(string, len); + warning(_("ignoring unknown core.fsync component '%s'"), component); + free(component); + } + +next_name: + string = ep; + } + + return (current & ~negative) | positive; +} + int git_parse_maybe_bool(const char *value) { int v = git_parse_maybe_bool_text(value); @@ -1600,6 +1667,13 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.fsync")) { + if (!value) + return config_error_nonbool(var); + fsync_components = parse_fsync_components(var, value); + return 0; + } + if (!strcmp(var, "core.fsyncmethod")) { if (!value) return config_error_nonbool(var); @@ -1613,6 +1687,8 @@ static int git_default_core_config(const char *var, const char *value, void *cb) } if (!strcmp(var, "core.fsyncobjectfiles")) { + if (fsync_object_files < 0) + warning(_("core.fsyncobjectfiles is deprecated; use core.fsync instead")); fsync_object_files = git_config_bool(var, value); return 0; } diff --git a/environment.c b/environment.c index 36ca5fb2e77..698f03a2f47 100644 --- a/environment.c +++ b/environment.c @@ -42,7 +42,7 @@ const char *git_attributes_file; const char *git_hooks_path; int zlib_compression_level = Z_BEST_SPEED; int pack_compression_level = Z_DEFAULT_COMPRESSION; -int fsync_object_files; +int fsync_object_files = -1; int use_fsync = -1; enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT; enum fsync_component fsync_components = FSYNC_COMPONENTS_DEFAULT; From patchwork Thu Mar 10 22:43:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12776986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0EC8C433F5 for ; Thu, 10 Mar 2022 22:43:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344434AbiCJWoo (ORCPT ); Thu, 10 Mar 2022 17:44:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239006AbiCJWof (ORCPT ); Thu, 10 Mar 2022 17:44:35 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 652A2D1087 for ; Thu, 10 Mar 2022 14:43:33 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id u10so10278389wra.9 for ; Thu, 10 Mar 2022 14:43:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8LMEDrdKX+poM1jfU5csXLomxtL/nPjVddZz+EGfo5E=; b=pbsgh//60xtZ6tSNj+B60fOBMwjU56hamz+nz5thOl4xJbHR3M7lZviP1rwFt+5qMl rz8kt/1+6Jg6B2e1Ae5KzKXlV9SIu1FDWPbvizsGDNcpxr9AONOt7bTDxUrqrFYq7khg +a5GtGfN2UhzkfyCelslFwjY4ueRo4HMG14+MUsYv6x+OvGUXPGshptBIg4DkGtec16Z YGj3MlVQB4jHA0xnvL9bCE2ObGq0t8mPnHpeaMpJmFe7IxCC/0jkXazBTsQZyFelqtVC kCJoEmXfsmDPSdulkEv0srCOfpgmipoRLmhYctw3kqOteXCdhDaZDvgta6mES61Y28NX dyfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8LMEDrdKX+poM1jfU5csXLomxtL/nPjVddZz+EGfo5E=; b=qZEQRpSv64DhFmDaI5ph+DjT9sJCfwMsUs++Umcbj6/yg6IFkrqEWnDICcq4Lc2Abe z08ezaBICYkR7mB54tCbGaOblm1xBO55f9H0EWbnlnYShRSkruKIKs/xInXNASSI65dC kcnpeqQg6yn7dWyUM/L/HxdN9mMUsFvwSM0BhJOzFIe6Opz3zxqSzGjcAlQtd5TALuj0 QYDnIenTPSz6E0Z1V2fTb0kpqqRHasgfp2NwpaH/YPERqSyo7oJSqRpe87TraZT8tqPE /fYoX4nvZZfgUgJSjYgErb1VryA4EsnJABmqckzjQTSDYHPTIfdPCaT8PCzp8arH3v7M 4O6g== X-Gm-Message-State: AOAM531Pz1tslWEfKsKta3dczOVTbR20rYgEsU1WLRFL2Q4Xy2Y/XSyK ufyJERtakR/ORmp72inSneEm1Jc4Lxs= X-Google-Smtp-Source: ABdhPJx6fxLFJ9pKIiSbDb9SNI3wjMzUbyZw99422GfEuKXlAj9XV7OZYYuGY9NZF4Fq4VZqGt1JpA== X-Received: by 2002:adf:f1c1:0:b0:1e3:1381:7bd5 with SMTP id z1-20020adff1c1000000b001e313817bd5mr5210922wro.450.1646952211858; Thu, 10 Mar 2022 14:43:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v14-20020adfd18e000000b0020373e5319asm5626360wrc.103.2022.03.10.14.43.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 14:43:31 -0800 (PST) Message-Id: <757f6d0bbd21550a9e8f4783eb0b3c351b12f930.1646952205.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 10 Mar 2022 22:43:23 +0000 Subject: [PATCH v6 5/6] core.fsync: new option to harden the index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, Johannes Schindelin , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit introduces the new ability for the user to harden the index. In the event of a system crash, the index must be durable for the user to actually find a file that has been added to the repo and then deleted from the working tree. We use the presence of the COMMIT_LOCK flag and absence of the alternate_index_output as a proxy for determining whether we're updating the persistent index of the repo or some temporary index. We don't sync these temporary indexes. Signed-off-by: Neeraj Singh --- cache.h | 1 + config.c | 1 + read-cache.c | 19 +++++++++++++------ 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/cache.h b/cache.h index 7ac1959258d..e08eeac6c15 100644 --- a/cache.h +++ b/cache.h @@ -1004,6 +1004,7 @@ enum fsync_component { FSYNC_COMPONENT_PACK = 1 << 1, FSYNC_COMPONENT_PACK_METADATA = 1 << 2, FSYNC_COMPONENT_COMMIT_GRAPH = 1 << 3, + FSYNC_COMPONENT_INDEX = 1 << 4, }; #define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ diff --git a/config.c b/config.c index 94a4598b5fc..80f33c91982 100644 --- a/config.c +++ b/config.c @@ -1331,6 +1331,7 @@ static const struct fsync_component_name { { "pack", FSYNC_COMPONENT_PACK }, { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, + { "index", FSYNC_COMPONENT_INDEX }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string) diff --git a/read-cache.c b/read-cache.c index df869691fd4..7683b679258 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2842,7 +2842,7 @@ static int record_ieot(void) * rely on it. */ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, - int strip_extensions) + int strip_extensions, unsigned flags) { uint64_t start = getnanotime(); struct hashfile *f; @@ -2856,6 +2856,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, struct strbuf previous_name_buf = STRBUF_INIT, *previous_name; int drop_cache_tree = istate->drop_cache_tree; off_t offset; + int csum_fsync_flag; int ieot_entries = 1; struct index_entry_offset_table *ieot = NULL; int nr, nr_threads; @@ -3089,7 +3090,13 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, return -1; } - finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_NONE, CSUM_HASH_IN_STREAM); + csum_fsync_flag = 0; + if (!alternate_index_output && (flags & COMMIT_LOCK)) + csum_fsync_flag = CSUM_FSYNC; + + finalize_hashfile(f, istate->oid.hash, FSYNC_COMPONENT_INDEX, + CSUM_HASH_IN_STREAM | csum_fsync_flag); + if (close_tempfile_gently(tempfile)) { error(_("could not close '%s'"), get_tempfile_path(tempfile)); return -1; @@ -3144,7 +3151,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l */ trace2_region_enter_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); - ret = do_write_index(istate, lock->tempfile, 0); + ret = do_write_index(istate, lock->tempfile, 0, flags); trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); @@ -3238,7 +3245,7 @@ static int clean_shared_index_files(const char *current_hex) } static int write_shared_index(struct index_state *istate, - struct tempfile **temp) + struct tempfile **temp, unsigned flags) { struct split_index *si = istate->split_index; int ret, was_full = !istate->sparse_index; @@ -3248,7 +3255,7 @@ static int write_shared_index(struct index_state *istate, trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); - ret = do_write_index(si->base, *temp, 1); + ret = do_write_index(si->base, *temp, 1, flags); trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3357,7 +3364,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret = do_write_locked_index(istate, lock, flags); goto out; } - ret = write_shared_index(istate, &temp); + ret = write_shared_index(istate, &temp, flags); saved_errno = errno; if (is_tempfile_active(temp)) From patchwork Thu Mar 10 22:43:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Neeraj Singh (WINDOWS-SFS)" X-Patchwork-Id: 12776988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAC2EC433EF for ; Thu, 10 Mar 2022 22:43:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344404AbiCJWot (ORCPT ); Thu, 10 Mar 2022 17:44:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344420AbiCJWog (ORCPT ); Thu, 10 Mar 2022 17:44:36 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73290D2276 for ; Thu, 10 Mar 2022 14:43:34 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id u10so10278434wra.9 for ; Thu, 10 Mar 2022 14:43:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=kEvQF8GZwqNj9QVImtLId8cF12bFTehdlgQQfVzDGQ4=; b=KAr24GH2aZmDMtArA7GQZn/dcKTN9LphBYU3DH+GDpe60G3DPc+UoT2JgyFqgGu3na 5boakKQ4JV0UGDSPOFyv1XyCKhk6nySPCfgMHcssMG8bju2K0eS1MHlT8yzb+VKlNeaP incBn/PKKu+7Cnb/FrfQ9b1fbQTavZ/B8PskSBpPqBooytOvZ2ZdwHMWxwRfR64M1PFE Gb1ENk+RcToZ5SY7KvDGjiaP8nlNejfIqMIExeu1FzcL/1/5pEEA/cqTNpzGaiqIKMq1 4Tlzz0i54GNtuWEYO6RGF5KfAB53T5k8NSLrE8LXwNeR8sU4PnYbPQfrQzW66pal/o8X qrTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=kEvQF8GZwqNj9QVImtLId8cF12bFTehdlgQQfVzDGQ4=; b=FxY1u8YN49bBxo+gbgnShpAiBcAFksO8tFvF/c6G5S1yI35nc179XMFvx7Yakse5ad zBsO8wPaOcK2QHohKOMNiN7YGe6zLgBUr3J6/4ZC6eTVKmxVgcUfJ+JfeYbbNyGbDefz krVtKp/nKeDbDCc1RYHvYHkTCsqG2dpvNoQ1bmAR/K48w27lcnoaiuchFqfRH0NXfpUw uimlZy6XZqzWTN/Pvgk4yd5NYv+AM9zEA0hg6hQ3cz7aTa6QBZYrawLY3RBn4X+/3Dhk 8yNHjrogyLFTSAtbvyz2Oc4+zGrNNajjwvbLV1zt42k46AhTu28VyQG9vPo1Y9Zpzewz 0uvw== X-Gm-Message-State: AOAM532QXY3cimlxs8p2HL3F58fETAkzdTtfjm8yjYKQFG5pa8fEB+qg ADi8vmQ53FHzy98K3O+r4DlLRHVeny8= X-Google-Smtp-Source: ABdhPJz1giO/ozzYWLTVfx4ffFd1tTZJODU2e9bB3gh1C9ZanZ2FsJ7Ctj84N4JEyYxfHb8EwoUAAg== X-Received: by 2002:a5d:64e5:0:b0:1fb:5b9a:1a20 with SMTP id g5-20020a5d64e5000000b001fb5b9a1a20mr5091012wri.34.1646952212821; Thu, 10 Mar 2022 14:43:32 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v12-20020a5d4a4c000000b001e68ba61747sm5154996wrs.16.2022.03.10.14.43.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Mar 2022 14:43:32 -0800 (PST) Message-Id: <7e4cc6e10a5d88f4c6c44efaa68f2325007fd935.1646952205.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 10 Mar 2022 22:43:24 +0000 Subject: [PATCH v6 6/6] core.fsync: documentation and user-friendly aggregate options Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: rsbecker@nexbridge.com, bagasdotme@gmail.com, newren@gmail.com, avarab@gmail.com, nksingh85@gmail.com, ps@pks.im, sandals@crustytoothpaste.net, Johannes Schindelin , "Neeraj K. Singh" , Neeraj Singh Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Neeraj Singh From: Neeraj Singh This commit adds aggregate options for the core.fsync setting that are more user-friendly. These options are specified in terms of 'levels of safety', indicating which Git operations are considered to be sync points for durability. The new documentation is also included here in its entirety for ease of review. Signed-off-by: Neeraj Singh --- Documentation/config/core.txt | 40 +++++++++++++++++++++++++++++++++++ cache.h | 23 +++++++++++++++++--- config.c | 5 +++++ 3 files changed, 65 insertions(+), 3 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index ab911d6e269..37105a7be40 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -547,6 +547,46 @@ core.whitespace:: is relevant for `indent-with-non-tab` and when Git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. +core.fsync:: + A comma-separated list of components of the repository that + should be hardened via the core.fsyncMethod when created or + modified. You can disable hardening of any component by + prefixing it with a '-'. Items that are not hardened may be + lost in the event of an unclean system shutdown. Unless you + have special requirements, it is recommended that you leave + this option empty or pick one of `committed`, `added`, + or `all`. ++ +When this configuration is encountered, the set of components starts with +the platform default value, disabled components are removed, and additional +components are added. `none` resets the state so that the platform default +is ignored. ++ +The empty string resets the fsync configuration to the platform +default. The platform default on most platform is equivalent to +`core.fsync=committed,-loose-object`, which has good performance, +but risks losing recent work in the event of an unclean system shutdown. ++ +* `none` clears the set of fsynced components. +* `loose-object` hardens objects added to the repo in loose-object form. +* `pack` hardens objects added to the repo in packfile form. +* `pack-metadata` hardens packfile bitmaps and indexes. +* `commit-graph` hardens the commit graph file. +* `index` hardens the index when it is modified. +* `objects` is an aggregate option that is equivalent to + `loose-object,pack`. +* `derived-metadata` is an aggregate option that is equivalent to + `pack-metadata,commit-graph`. +* `committed` is an aggregate option that is currently equivalent to + `objects`. This mode sacrifices some performance to ensure that work + that is committed to the repository with `git commit` or similar commands + is hardened. +* `added` is an aggregate option that is currently equivalent to + `committed,index`. This mode sacrifices additional performance to + ensure that the results of commands like `git add` and similar operations + are hardened. +* `all` is an aggregate option that syncs all individual components above. + core.fsyncMethod:: A value indicating the strategy Git will use to harden repository data using fsync and related primitives. diff --git a/cache.h b/cache.h index e08eeac6c15..86680f144ec 100644 --- a/cache.h +++ b/cache.h @@ -1007,9 +1007,26 @@ enum fsync_component { FSYNC_COMPONENT_INDEX = 1 << 4, }; -#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENT_PACK | \ - FSYNC_COMPONENT_PACK_METADATA | \ - FSYNC_COMPONENT_COMMIT_GRAPH) +#define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK) + +#define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH) + +#define FSYNC_COMPONENTS_DEFAULT (FSYNC_COMPONENTS_OBJECTS | \ + FSYNC_COMPONENTS_DERIVED_METADATA | \ + ~FSYNC_COMPONENT_LOOSE_OBJECT) + +#define FSYNC_COMPONENTS_COMMITTED (FSYNC_COMPONENTS_OBJECTS) + +#define FSYNC_COMPONENTS_ADDED (FSYNC_COMPONENTS_COMMITTED | \ + FSYNC_COMPONENT_INDEX) + +#define FSYNC_COMPONENTS_ALL (FSYNC_COMPONENT_LOOSE_OBJECT | \ + FSYNC_COMPONENT_PACK | \ + FSYNC_COMPONENT_PACK_METADATA | \ + FSYNC_COMPONENT_COMMIT_GRAPH | \ + FSYNC_COMPONENT_INDEX) /* * A bitmask indicating which components of the repo should be fsynced. diff --git a/config.c b/config.c index 80f33c91982..fd8e1659312 100644 --- a/config.c +++ b/config.c @@ -1332,6 +1332,11 @@ static const struct fsync_component_name { { "pack-metadata", FSYNC_COMPONENT_PACK_METADATA }, { "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH }, { "index", FSYNC_COMPONENT_INDEX }, + { "objects", FSYNC_COMPONENTS_OBJECTS }, + { "derived-metadata", FSYNC_COMPONENTS_DERIVED_METADATA }, + { "committed", FSYNC_COMPONENTS_COMMITTED }, + { "added", FSYNC_COMPONENTS_ADDED }, + { "all", FSYNC_COMPONENTS_ALL }, }; static enum fsync_component parse_fsync_components(const char *var, const char *string)