| Message ID | 20200916034307.2092020-1-nickrterrell@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Update to zstd-1.4.6 | expand |
> On Sep 15, 2020, at 8:42 PM, Nick Terrell <nickrterrell@gmail.com> wrote: > > From: Nick Terrell <terrelln@fb.com> > > Upgrade to the latest upstream zstd version 1.4.6. > > This patch is 100% generated from upstream zstd commit c4763f087c2b [0]. > > This patch is very large because it is transitioning from the custom > kernel zstd to using upstream directly. The new zstd follows upstreams > file structure which is different. Future update patches will be much > smaller because they will only contain the changes from one upstream > zstd release. > > The benefits of this patch are as follows: > 1. Using upstream directly with automated script to generate kernel > code. This allows us to update the kernel every upstream release, so > the kernel gets the latest bug fixes and performance improvements, > and doesn't get 3 years out of date again. The automation and the > translated code are tested every upstream commit to ensure it > continues to work. > 2. Upgrades from a custom zstd based on 1.3.1 to 1.4.6, getting 3 years > of performance improvements and bug fixes. On x86_64 I've measured > 15% faster BtrFS and SquashFS decompression+read speeds, 35% faster > kernel decompression, and 30% faster ZRAM decompression+read speeds. > Additionally, the latest zstd uses ~1 KB less stack space for > compression. > 3. Switches to using the upstream API directly. It is slightly less > ergonomic for the kernel use case, where malloc/free aren't provided. > But, it means that users don't need to familiarize themselves with 2 > zstd APIs. > > I chose the bulk update instead of replaying upstream commits because > there have been ~3500 upstream commits since the 1.3.1 release, zstd > wasn't ready to be used in the kernel as-is before a month ago, and not > all upstream zstd commits build. The bulk update preserves bisectablity > because bugs can be bisected to the zstd version update. At that point > the update can be reverted, and we can work with upstream to find and > fix the bug. > > Note that upstream zstd release 1.4.6 doesn't exist yet. I have cut a > staging branch at c4763f087c2b [0] and will apply any changes requested > to the staging branch. Once we're ready to merge this update I will cut > a zstd release at the commit we merge, so we have a known zstd release > in the kernel. > > [0] https://github.com/facebook/zstd/commit/c4763f087c2b4b5857a8323ff3360b240db23786 > > Signed-off-by: Nick Terrell <terrelln@fb.com> Below is a diff that shows the difference between upstream zstd imported directly into the kernel, and the version in this patch that uses upstreams automation generate a working zstd. I hope it is helpful for review, since I know the full patch is way to large for a meaningful review. The automation does several necessary things: * Rewrite libc headers * Replace bundled xxhash with kernel xxhash * Provide zstd_deps.h, which holds all of zstd’s libc dependencies It also hardwires certain preprocessor macros to avoid unnecessary portability code in the kernel. This is not strictly necessary, because these macros could be defined at compile time. See [0] for a list of macros. This diff is also available at [0]. [0] https://gist.github.com/terrelln/5a266ef4f6ee8bc60dde192daaaf2c97 [1] https://github.com/facebook/zstd/blob/d96e98cfde66e9e20dcadcfd9ed3b82ba648adfe/contrib/linux-kernel/Makefile#L17 Best, Nick --- include/linux/zstd.h | 28 +--- include/linux/zstd_errors.h | 24 +-- lib/zstd/common/bitstream.h | 28 +--- lib/zstd/common/compiler.h | 91 ++--------- lib/zstd/common/cpu.h | 21 +-- lib/zstd/common/debug.h | 6 - lib/zstd/common/entropy_common.c | 7 +- lib/zstd/common/error_private.h | 18 +-- lib/zstd/common/fse.h | 8 +- lib/zstd/common/fse_decompress.c | 13 -- lib/zstd/common/huf.h | 8 +- lib/zstd/common/mem.h | 77 +--------- lib/zstd/common/zstd_deps.h | 110 ++++++------- lib/zstd/common/zstd_internal.h | 35 +---- lib/zstd/compress/fse_compress.c | 80 ---------- lib/zstd/compress/hist.c | 16 -- lib/zstd/compress/huf_compress.c | 35 ----- lib/zstd/compress/zstd_compress.c | 135 +--------------- lib/zstd/compress/zstd_compress_internal.h | 42 +---- lib/zstd/compress/zstd_compress_superblock.h | 2 +- lib/zstd/compress/zstd_cwksp.h | 6 - lib/zstd/compress/zstd_double_fast.h | 6 - lib/zstd/compress/zstd_fast.h | 6 - lib/zstd/compress/zstd_lazy.c | 4 +- lib/zstd/compress/zstd_lazy.h | 6 - lib/zstd/compress/zstd_ldm.h | 8 +- lib/zstd/compress/zstd_opt.h | 6 - lib/zstd/decompress/huf_decompress.c | 145 ------------------ lib/zstd/decompress/zstd_ddict.c | 3 - lib/zstd/decompress/zstd_ddict.h | 2 +- lib/zstd/decompress/zstd_decompress.c | 81 +--------- lib/zstd/decompress/zstd_decompress_block.c | 4 +- lib/zstd/decompress/zstd_decompress_block.h | 2 +- .../decompress/zstd_decompress_internal.h | 2 +- 34 files changed, 104 insertions(+), 961 deletions(-) diff --git a/include/linux/zstd.h b/include/linux/zstd.h index 4e9fd93bd386..cf4bd96806d5 100644 --- a/include/linux/zstd.h +++ b/include/linux/zstd.h @@ -7,33 +7,18 @@ * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ -#if defined (__cplusplus) -extern "C" { -#endif #ifndef ZSTD_H_235446 #define ZSTD_H_235446 /* ====== Dependency ======*/ -#include <limits.h> /* INT_MAX */ -#include <stddef.h> /* size_t */ +#include <linux/limits.h> /* INT_MAX */ +#include <linux/types.h> /* size_t */ /* ===== ZSTDLIB_API : control library symbols visibility ===== */ -#ifndef ZSTDLIB_VISIBILITY -# if defined(__GNUC__) && (__GNUC__ >= 4) -# define ZSTDLIB_VISIBILITY __attribute__ ((visibility ("default"))) -# else -# define ZSTDLIB_VISIBILITY -# endif -#endif -#if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1) -# define ZSTDLIB_API __declspec(dllexport) ZSTDLIB_VISIBILITY -#elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1) -# define ZSTDLIB_API __declspec(dllimport) ZSTDLIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ -#else -# define ZSTDLIB_API ZSTDLIB_VISIBILITY -#endif +#define ZSTDLIB_VISIBILITY +#define ZSTDLIB_API ZSTDLIB_VISIBILITY /******************************************************************************* @@ -1037,7 +1022,7 @@ ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict); * Use them only in association with static linking. * ***************************************************************************************/ -#if defined(ZSTD_STATIC_LINKING_ONLY) && !defined(ZSTD_H_ZSTD_STATIC_LINKING_ONLY) +#if !defined(ZSTD_H_ZSTD_STATIC_LINKING_ONLY) #define ZSTD_H_ZSTD_STATIC_LINKING_ONLY /**************************************************************************************** @@ -2115,6 +2100,3 @@ ZSTDLIB_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, #endif /* ZSTD_H_ZSTD_STATIC_LINKING_ONLY */ -#if defined (__cplusplus) -} -#endif diff --git a/include/linux/zstd_errors.h b/include/linux/zstd_errors.h index 998398e7e57f..5d0688650b04 100644 --- a/include/linux/zstd_errors.h +++ b/include/linux/zstd_errors.h @@ -11,29 +11,14 @@ #ifndef ZSTD_ERRORS_H_398273423 #define ZSTD_ERRORS_H_398273423 -#if defined (__cplusplus) -extern "C" { -#endif /*===== dependency =====*/ -#include <stddef.h> /* size_t */ +#include <linux/types.h> /* size_t */ /* ===== ZSTDERRORLIB_API : control library symbols visibility ===== */ -#ifndef ZSTDERRORLIB_VISIBILITY -# if defined(__GNUC__) && (__GNUC__ >= 4) -# define ZSTDERRORLIB_VISIBILITY __attribute__ ((visibility ("default"))) -# else -# define ZSTDERRORLIB_VISIBILITY -# endif -#endif -#if defined(ZSTD_DLL_EXPORT) && (ZSTD_DLL_EXPORT==1) -# define ZSTDERRORLIB_API __declspec(dllexport) ZSTDERRORLIB_VISIBILITY -#elif defined(ZSTD_DLL_IMPORT) && (ZSTD_DLL_IMPORT==1) -# define ZSTDERRORLIB_API __declspec(dllimport) ZSTDERRORLIB_VISIBILITY /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/ -#else -# define ZSTDERRORLIB_API ZSTDERRORLIB_VISIBILITY -#endif +#define ZSTDERRORLIB_VISIBILITY +#define ZSTDERRORLIB_API ZSTDERRORLIB_VISIBILITY /*-********************************************* * Error codes list @@ -87,8 +72,5 @@ ZSTDERRORLIB_API ZSTD_ErrorCode ZSTD_getErrorCode(size_t functionResult); ZSTDERRORLIB_API const char* ZSTD_getErrorString(ZSTD_ErrorCode code); /**< Same as ZSTD_getErrorName, but using a `ZSTD_ErrorCode` enum argument */ -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_ERRORS_H_398273423 */ diff --git a/lib/zstd/common/bitstream.h b/lib/zstd/common/bitstream.h index c74918502c5c..e11f95e5bb5e 100644 --- a/lib/zstd/common/bitstream.h +++ b/lib/zstd/common/bitstream.h @@ -14,9 +14,6 @@ #ifndef BITSTREAM_H_MODULE #define BITSTREAM_H_MODULE -#if defined (__cplusplus) -extern "C" { -#endif /* * This API consists of small unitary functions, which must be inlined for best performance. * Since link-time-optimization is not available for all compilers, @@ -35,13 +32,6 @@ extern "C" { /*========================================= * Target specific =========================================*/ -#ifndef ZSTD_NO_INTRINSICS -# if defined(__BMI__) && defined(__GNUC__) -# include <immintrin.h> /* support for bextr (experimental) */ -# elif defined(__ICCARM__) -# include <intrinsics.h> -# endif -#endif #define STREAM_ACCUMULATOR_MIN_32 25 #define STREAM_ACCUMULATOR_MIN_64 57 @@ -141,17 +131,8 @@ MEM_STATIC unsigned BIT_highbit32 (U32 val) { assert(val != 0); { -# if defined(_MSC_VER) /* Visual */ -# if STATIC_BMI2 == 1 - return _lzcnt_u32(val) ^ 31; -# else - unsigned long r = 0; - return _BitScanReverse(&r, val) ? (unsigned)r : 0; -# endif -# elif defined(__GNUC__) && (__GNUC__ >= 3) /* Use GCC Intrinsic */ +# if (__GNUC__ >= 3) /* Use GCC Intrinsic */ return __builtin_clz (val) ^ 31; -# elif defined(__ICCARM__) /* IAR Intrinsic */ - return 31 - __CLZ(val); # else /* Software version */ static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, @@ -337,12 +318,8 @@ MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getMiddleBits(size_t bitContainer, U32 c MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits) { -#if defined(STATIC_BMI2) && STATIC_BMI2 == 1 - return _bzhi_u64(bitContainer, nbBits); -#else assert(nbBits < BIT_MASK_SIZE); return bitContainer & BIT_mask[nbBits]; -#endif } /*! BIT_lookBits() : @@ -456,8 +433,5 @@ MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* DStream) return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer)*8)); } -#if defined (__cplusplus) -} -#endif #endif /* BITSTREAM_H_MODULE */ diff --git a/lib/zstd/common/compiler.h b/lib/zstd/common/compiler.h index 4c65ccbb5b69..63b70944df2f 100644 --- a/lib/zstd/common/compiler.h +++ b/lib/zstd/common/compiler.h @@ -16,38 +16,21 @@ *********************************************************/ /* force inlining */ -#if !defined(ZSTD_NO_INLINE) #if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */ # define INLINE_KEYWORD inline #else # define INLINE_KEYWORD #endif -#if defined(__GNUC__) || defined(__ICCARM__) -# define FORCE_INLINE_ATTR __attribute__((always_inline)) -#elif defined(_MSC_VER) -# define FORCE_INLINE_ATTR __forceinline -#else -# define FORCE_INLINE_ATTR -#endif - -#else - -#define INLINE_KEYWORD -#define FORCE_INLINE_ATTR +#define FORCE_INLINE_ATTR __attribute__((always_inline)) -#endif /** On MSVC qsort requires that functions passed into it use the __cdecl calling conversion(CC). This explictly marks such functions as __cdecl so that the code will still compile if a CC other than __cdecl has been made the default. */ -#if defined(_MSC_VER) -# define WIN_CDECL __cdecl -#else -# define WIN_CDECL -#endif +#define WIN_CDECL /** * FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant @@ -73,32 +56,16 @@ #endif /* UNUSED_ATTR tells the compiler it is okay if the function is unused. */ -#if defined(__GNUC__) -# define UNUSED_ATTR __attribute__((unused)) -#else -# define UNUSED_ATTR -#endif +#define UNUSED_ATTR __attribute__((unused)) /* force no inlining */ -#ifdef _MSC_VER -# define FORCE_NOINLINE static __declspec(noinline) -#else -# if defined(__GNUC__) || defined(__ICCARM__) -# define FORCE_NOINLINE static __attribute__((__noinline__)) -# else -# define FORCE_NOINLINE static -# endif -#endif +#define FORCE_NOINLINE static __attribute__((__noinline__)) /* target attribute */ #ifndef __has_attribute #define __has_attribute(x) 0 /* Compatibility with non-clang compilers. */ #endif -#if defined(__GNUC__) || defined(__ICCARM__) -# define TARGET_ATTRIBUTE(target) __attribute__((__target__(target))) -#else -# define TARGET_ATTRIBUTE(target) -#endif +#define TARGET_ATTRIBUTE(target) __attribute__((__target__(target))) /* Enable runtime BMI2 dispatch based on the CPU. * Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default. @@ -117,24 +84,15 @@ /* prefetch * can be disabled, by declaring NO_PREFETCH build macro */ -#if defined(NO_PREFETCH) -# define PREFETCH_L1(ptr) (void)(ptr) /* disabled */ -# define PREFETCH_L2(ptr) (void)(ptr) /* disabled */ +#if ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) +# define PREFETCH_L1(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */) +# define PREFETCH_L2(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 2 /* locality */) +#elif defined(__aarch64__) +# define PREFETCH_L1(ptr) __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr))) +# define PREFETCH_L2(ptr) __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr))) #else -# if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) /* _mm_prefetch() is not defined outside of x86/x64 */ -# include <mmintrin.h> /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */ -# define PREFETCH_L1(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0) -# define PREFETCH_L2(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T1) -# elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) -# define PREFETCH_L1(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */) -# define PREFETCH_L2(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 2 /* locality */) -# elif defined(__aarch64__) -# define PREFETCH_L1(ptr) __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr))) -# define PREFETCH_L2(ptr) __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr))) -# else -# define PREFETCH_L1(ptr) (void)(ptr) /* disabled */ -# define PREFETCH_L2(ptr) (void)(ptr) /* disabled */ -# endif +# define PREFETCH_L1(ptr) (void)(ptr) /* disabled */ +# define PREFETCH_L2(ptr) (void)(ptr) /* disabled */ #endif /* NO_PREFETCH */ #define CACHELINE_SIZE 64 @@ -165,35 +123,12 @@ * If you can remove a LIKELY/UNLIKELY annotation without speed changes in gcc * and clang, please do. */ -#if defined(__GNUC__) #define LIKELY(x) (__builtin_expect((x), 1)) #define UNLIKELY(x) (__builtin_expect((x), 0)) -#else -#define LIKELY(x) (x) -#define UNLIKELY(x) (x) -#endif /* disable warnings */ -#ifdef _MSC_VER /* Visual Studio */ -# include <intrin.h> /* For Visual 2005 */ -# pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */ -# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ -# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */ -# pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */ -# pragma warning(disable : 4324) /* disable: C4324: padded structure */ -#endif /*Like DYNAMIC_BMI2 but for compile time determination of BMI2 support*/ -#ifndef STATIC_BMI2 -# if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) -# ifdef __AVX2__ //MSVC does not have a BMI2 specific flag, but every CPU that supports AVX2 also supports BMI2 -# define STATIC_BMI2 1 -# endif -# endif -#endif -#ifndef STATIC_BMI2 - #define STATIC_BMI2 0 -#endif #endif /* ZSTD_COMPILER_H */ diff --git a/lib/zstd/common/cpu.h b/lib/zstd/common/cpu.h index cb210593ead9..ab9aa33b3e50 100644 --- a/lib/zstd/common/cpu.h +++ b/lib/zstd/common/cpu.h @@ -18,9 +18,6 @@ #include "mem.h" -#ifdef _MSC_VER -#include <intrin.h> -#endif typedef struct { U32 f1c; @@ -34,23 +31,7 @@ MEM_STATIC ZSTD_cpuid_t ZSTD_cpuid(void) { U32 f1d = 0; U32 f7b = 0; U32 f7c = 0; -#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86)) - int reg[4]; - __cpuid((int*)reg, 0); - { - int const n = reg[0]; - if (n >= 1) { - __cpuid((int*)reg, 1); - f1c = (U32)reg[2]; - f1d = (U32)reg[3]; - } - if (n >= 7) { - __cpuidex((int*)reg, 7, 0); - f7b = (U32)reg[1]; - f7c = (U32)reg[2]; - } - } -#elif defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__) +#if defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__) /* The following block like the normal cpuid branch below, but gcc * reserves ebx for use of its pic register so we must specially * handle the save and restore to avoid clobbering the register diff --git a/lib/zstd/common/debug.h b/lib/zstd/common/debug.h index 8b5734366ca2..d6b2f7b9213b 100644 --- a/lib/zstd/common/debug.h +++ b/lib/zstd/common/debug.h @@ -32,9 +32,6 @@ #ifndef DEBUG_H_12987983217 #define DEBUG_H_12987983217 -#if defined (__cplusplus) -extern "C" { -#endif /* static assert is triggered at compile time, leaving no runtime artefact. @@ -100,8 +97,5 @@ extern int g_debuglevel; /* the variable is only declared, #endif -#if defined (__cplusplus) -} -#endif #endif /* DEBUG_H_12987983217 */ diff --git a/lib/zstd/common/entropy_common.c b/lib/zstd/common/entropy_common.c index b1213a964dc1..d97a24d46bdf 100644 --- a/lib/zstd/common/entropy_common.c +++ b/lib/zstd/common/entropy_common.c @@ -42,13 +42,8 @@ static U32 FSE_ctz(U32 val) { assert(val != 0); { -# if defined(_MSC_VER) /* Visual */ - unsigned long r=0; - return _BitScanForward(&r, val) ? (unsigned)r : 0; -# elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */ +# if (__GNUC__ >= 3) /* GCC Intrinsic */ return __builtin_ctz(val); -# elif defined(__ICCARM__) /* IAR Intrinsic */ - return __CTZ(val); # else /* Software version */ U32 count = 0; while ((val & 1) == 0) { diff --git a/lib/zstd/common/error_private.h b/lib/zstd/common/error_private.h index 71b37b8dfa43..17f5563c6e93 100644 --- a/lib/zstd/common/error_private.h +++ b/lib/zstd/common/error_private.h @@ -13,30 +13,19 @@ #ifndef ERROR_H_MODULE #define ERROR_H_MODULE -#if defined (__cplusplus) -extern "C" { -#endif /* **************************************** * Dependencies ******************************************/ #include "zstd_deps.h" /* size_t */ -#include "zstd_errors.h" /* enum list */ +#include <linux/zstd_errors.h> /* enum list */ /* **************************************** * Compiler-specific ******************************************/ -#if defined(__GNUC__) -# define ERR_STATIC static __attribute__((unused)) -#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) -# define ERR_STATIC static inline -#elif defined(_MSC_VER) -# define ERR_STATIC static __inline -#else -# define ERR_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */ -#endif +#define ERR_STATIC static __attribute__((unused)) /*-**************************************** @@ -73,8 +62,5 @@ ERR_STATIC const char* ERR_getErrorName(size_t code) return ERR_getErrorString(ERR_getErrorCode(code)); } -#if defined (__cplusplus) -} -#endif #endif /* ERROR_H_MODULE */ diff --git a/lib/zstd/common/fse.h b/lib/zstd/common/fse.h index 83a07847aaa0..d9c4abab0459 100644 --- a/lib/zstd/common/fse.h +++ b/lib/zstd/common/fse.h @@ -12,9 +12,6 @@ * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ -#if defined (__cplusplus) -extern "C" { -#endif #ifndef FSE_H #define FSE_H @@ -289,7 +286,7 @@ If there is an error, the function will return an error code, which can be teste #endif /* FSE_H */ -#if defined(FSE_STATIC_LINKING_ONLY) && !defined(FSE_H_FSE_STATIC_LINKING_ONLY) +#if !defined(FSE_H_FSE_STATIC_LINKING_ONLY) #define FSE_H_FSE_STATIC_LINKING_ONLY /* *** Dependency *** */ @@ -710,6 +707,3 @@ MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr) #endif /* FSE_STATIC_LINKING_ONLY */ -#if defined (__cplusplus) -} -#endif diff --git a/lib/zstd/common/fse_decompress.c b/lib/zstd/common/fse_decompress.c index c164430f9905..fdf46a4efca7 100644 --- a/lib/zstd/common/fse_decompress.c +++ b/lib/zstd/common/fse_decompress.c @@ -375,19 +375,6 @@ size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, typedef FSE_DTable DTable_max_t[FSE_DTABLE_SIZE_U32(FSE_MAX_TABLELOG)]; -#ifndef ZSTD_NO_UNUSED_FUNCTIONS -size_t FSE_buildDTable(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog) { - U32 wksp[FSE_BUILD_DTABLE_WKSP_SIZE_U32(FSE_TABLELOG_ABSOLUTE_MAX, FSE_MAX_SYMBOL_VALUE)]; - return FSE_buildDTable_wksp(dt, normalizedCounter, maxSymbolValue, tableLog, wksp, sizeof(wksp)); -} - -size_t FSE_decompress(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize) -{ - /* Static analyzer seems unable to understand this table will be properly initialized later */ - U32 wksp[FSE_DECOMPRESS_WKSP_SIZE_U32(FSE_MAX_TABLELOG, FSE_MAX_SYMBOL_VALUE)]; - return FSE_decompress_wksp(dst, dstCapacity, cSrc, cSrcSize, FSE_MAX_TABLELOG, wksp, sizeof(wksp)); -} -#endif #endif /* FSE_COMMONDEFS_ONLY */ diff --git a/lib/zstd/common/huf.h b/lib/zstd/common/huf.h index 772b11f16c34..3cb03211452f 100644 --- a/lib/zstd/common/huf.h +++ b/lib/zstd/common/huf.h @@ -12,9 +12,6 @@ * You may select, at your option, one of the above-listed licenses. ****************************************************************** */ -#if defined (__cplusplus) -extern "C" { -#endif #ifndef HUF_H_298734234 #define HUF_H_298734234 @@ -106,7 +103,7 @@ HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity, * because they are not guaranteed to remain stable in the future. * Only consider them in association with static linking. * *****************************************************************/ -#if defined(HUF_STATIC_LINKING_ONLY) && !defined(HUF_H_HUF_STATIC_LINKING_ONLY) +#if !defined(HUF_H_HUF_STATIC_LINKING_ONLY) #define HUF_H_HUF_STATIC_LINKING_ONLY /* *** Dependencies *** */ @@ -353,6 +350,3 @@ size_t HUF_readDTableX1_wksp_bmi2(HUF_DTable* DTable, const void* src, size_t sr #endif /* HUF_STATIC_LINKING_ONLY */ -#if defined (__cplusplus) -} -#endif diff --git a/lib/zstd/common/mem.h b/lib/zstd/common/mem.h index 32da28b85578..b4c64901d947 100644 --- a/lib/zstd/common/mem.h +++ b/lib/zstd/common/mem.h @@ -11,9 +11,6 @@ #ifndef MEM_H_MODULE #define MEM_H_MODULE -#if defined (__cplusplus) -extern "C" { -#endif /*-**************************************** * Dependencies @@ -24,19 +21,7 @@ extern "C" { /*-**************************************** * Compiler specifics ******************************************/ -#if defined(_MSC_VER) /* Visual Studio */ -# include <stdlib.h> /* _byteswap_ulong */ -# include <intrin.h> /* _byteswap_* */ -#endif -#if defined(__GNUC__) -# define MEM_STATIC static __inline __attribute__((unused)) -#elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) -# define MEM_STATIC static inline -#elif defined(_MSC_VER) -# define MEM_STATIC static __inline -#else -# define MEM_STATIC static /* this version may generate warnings for unused static functions; disable the relevant warning */ -#endif +#define MEM_STATIC static __inline __attribute__((unused)) #ifndef __has_builtin # define __has_builtin(x) 0 /* compat. with non-clang compilers */ @@ -137,13 +122,6 @@ void __asan_unpoison_memory_region(void const volatile *addr, size_t size); * See http://fastcompression.blogspot.fr/2015/08/accessing-unaligned-memory.html for details. * Prefer these methods in priority order (0 > 1 > 2) */ -#ifndef MEM_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ -# if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) ) -# define MEM_FORCE_MEMORY_ACCESS 2 -# elif defined(__INTEL_COMPILER) || defined(__GNUC__) || defined(__ICCARM__) -# define MEM_FORCE_MEMORY_ACCESS 1 -# endif -#endif MEM_STATIC unsigned MEM_32bits(void) { return sizeof(size_t)==4; } MEM_STATIC unsigned MEM_64bits(void) { return sizeof(size_t)==8; } @@ -154,47 +132,6 @@ MEM_STATIC unsigned MEM_isLittleEndian(void) return one.c[0]; } -#if defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==2) - -/* violates C standard, by lying on structure alignment. -Only use if no other choice to achieve best performance on target platform */ -MEM_STATIC U16 MEM_read16(const void* memPtr) { return *(const U16*) memPtr; } -MEM_STATIC U32 MEM_read32(const void* memPtr) { return *(const U32*) memPtr; } -MEM_STATIC U64 MEM_read64(const void* memPtr) { return *(const U64*) memPtr; } -MEM_STATIC size_t MEM_readST(const void* memPtr) { return *(const size_t*) memPtr; } - -MEM_STATIC void MEM_write16(void* memPtr, U16 value) { *(U16*)memPtr = value; } -MEM_STATIC void MEM_write32(void* memPtr, U32 value) { *(U32*)memPtr = value; } -MEM_STATIC void MEM_write64(void* memPtr, U64 value) { *(U64*)memPtr = value; } - -#elif defined(MEM_FORCE_MEMORY_ACCESS) && (MEM_FORCE_MEMORY_ACCESS==1) - -/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */ -/* currently only defined for gcc and icc */ -#if defined(_MSC_VER) || (defined(__INTEL_COMPILER) && defined(WIN32)) - __pragma( pack(push, 1) ) - typedef struct { U16 v; } unalign16; - typedef struct { U32 v; } unalign32; - typedef struct { U64 v; } unalign64; - typedef struct { size_t v; } unalignArch; - __pragma( pack(pop) ) -#else - typedef struct { U16 v; } __attribute__((packed)) unalign16; - typedef struct { U32 v; } __attribute__((packed)) unalign32; - typedef struct { U64 v; } __attribute__((packed)) unalign64; - typedef struct { size_t v; } __attribute__((packed)) unalignArch; -#endif - -MEM_STATIC U16 MEM_read16(const void* ptr) { return ((const unalign16*)ptr)->v; } -MEM_STATIC U32 MEM_read32(const void* ptr) { return ((const unalign32*)ptr)->v; } -MEM_STATIC U64 MEM_read64(const void* ptr) { return ((const unalign64*)ptr)->v; } -MEM_STATIC size_t MEM_readST(const void* ptr) { return ((const unalignArch*)ptr)->v; } - -MEM_STATIC void MEM_write16(void* memPtr, U16 value) { ((unalign16*)memPtr)->v = value; } -MEM_STATIC void MEM_write32(void* memPtr, U32 value) { ((unalign32*)memPtr)->v = value; } -MEM_STATIC void MEM_write64(void* memPtr, U64 value) { ((unalign64*)memPtr)->v = value; } - -#else /* default method, safe and standard. can sometimes prove slower */ @@ -234,13 +171,10 @@ MEM_STATIC void MEM_write64(void* memPtr, U64 value) ZSTD_memcpy(memPtr, &value, sizeof(value)); } -#endif /* MEM_FORCE_MEMORY_ACCESS */ MEM_STATIC U32 MEM_swap32(U32 in) { -#if defined(_MSC_VER) /* Visual Studio */ - return _byteswap_ulong(in); -#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \ +#if (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \ || (defined(__clang__) && __has_builtin(__builtin_bswap32)) return __builtin_bswap32(in); #else @@ -253,9 +187,7 @@ MEM_STATIC U32 MEM_swap32(U32 in) MEM_STATIC U64 MEM_swap64(U64 in) { -#if defined(_MSC_VER) /* Visual Studio */ - return _byteswap_uint64(in); -#elif (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \ +#if (defined (__GNUC__) && (__GNUC__ * 100 + __GNUC_MINOR__ >= 403)) \ || (defined(__clang__) && __has_builtin(__builtin_bswap64)) return __builtin_bswap64(in); #else @@ -411,8 +343,5 @@ MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val) } -#if defined (__cplusplus) -} -#endif #endif /* MEM_H_MODULE */ diff --git a/lib/zstd/common/zstd_deps.h b/lib/zstd/common/zstd_deps.h index 3b4db18765c5..d4ea0b2a4d4a 100644 --- a/lib/zstd/common/zstd_deps.h +++ b/lib/zstd/common/zstd_deps.h @@ -9,6 +9,7 @@ */ /* Need: + * NULL * ZSTD_memcpy() * ZSTD_memset() * ZSTD_memmove() @@ -20,62 +21,33 @@ * size_t * ptrdiff_t * INT_MAX - * ... + * UINT_MAX */ #ifndef ZSTD_DEPS_COMMON #define ZSTD_DEPS_COMMON -#include <limits.h> -#include <stddef.h> -#include <string.h> - -#if defined(__GNUC__) && __GNUC__ >= 4 -# define ZSTD_memcpy(d,s,l) __builtin_memcpy((d),(s),(l)) -# define ZSTD_memmove(d,s,l) __builtin_memmove((d),(s),(l)) -# define ZSTD_memset(p,v,l) __builtin_memset((p),(v),(l)) -#else -# define ZSTD_memcpy(d,s,l) memcpy((d),(s),(l)) -# define ZSTD_memmove(d,s,l) memmove((d),(s),(l)) -# define ZSTD_memset(p,v,l) memset((p),(v),(l)) -#endif - -/*-************************************************************** -* Basic Types -*****************************************************************/ -#if !defined (__VMS) && (defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) -# include <stdint.h> - typedef uint8_t BYTE; - typedef uint16_t U16; - typedef int16_t S16; - typedef uint32_t U32; - typedef int32_t S32; - typedef uint64_t U64; - typedef int64_t S64; -#else -# include <limits.h> -#if CHAR_BIT != 8 -# error "this implementation requires char to be exactly 8-bit type" -#endif - typedef unsigned char BYTE; -#if USHRT_MAX != 65535 -# error "this implementation requires short to be exactly 16-bit type" -#endif - typedef unsigned short U16; - typedef signed short S16; -#if UINT_MAX != 4294967295 -# error "this implementation requires int to be exactly 32-bit type" -#endif - typedef unsigned int U32; - typedef signed int S32; -/* note : there are no limits defined for long long type in C90. - * limits exist in C99, however, in such case, <stdint.h> is preferred */ - typedef unsigned long long U64; - typedef signed long long S64; -#endif +#include <linux/limits.h> +#include <linux/types.h> +#include <linux/stddef.h> + +typedef uint8_t BYTE; +typedef uint16_t U16; +typedef int16_t S16; +typedef uint32_t U32; +typedef int32_t S32; +typedef uint64_t U64; +typedef int64_t S64; + +#define ZSTD_memcpy(d,s,n) __builtin_memcpy((d),(s),(n)) +#define ZSTD_memmove(d,s,n) __builtin_memmove((d),(s),(n)) +#define ZSTD_memset(d,s,n) __builtin_memset((d),(s),(n)) #endif /* ZSTD_DEPS_COMMON */ -/* Need: +/* + * Define malloc as always failing. That means the user must + * either use ZSTD_customMem or statically allocate memory. + * Need: * ZSTD_malloc() * ZSTD_free() * ZSTD_calloc() @@ -84,11 +56,9 @@ #ifndef ZSTD_DEPS_MALLOC #define ZSTD_DEPS_MALLOC -#include <stdlib.h> - -#define ZSTD_malloc(s) malloc(s) -#define ZSTD_calloc(n,s) calloc((n), (s)) -#define ZSTD_free(p) free((p)) +#define ZSTD_malloc(s) (NULL) +#define ZSTD_free(p) ((void)0) +#define ZSTD_calloc(n,s) (NULL) #endif /* ZSTD_DEPS_MALLOC */ #endif /* ZSTD_DEPS_NEED_MALLOC */ @@ -102,39 +72,51 @@ #ifndef ZSTD_DEPS_MATH64 #define ZSTD_DEPS_MATH64 +#include <linux/math64.h> + static U64 ZSTD_div64(U64 dividend, U32 divisor) { - return dividend / divisor; + return div_u64(dividend, divisor); } #endif /* ZSTD_DEPS_MATH64 */ #endif /* ZSTD_DEPS_NEED_MATH64 */ -/* Need: +/* + * This is only requested when DEBUGLEVEL >= 1, meaning + * it is disabled in production. + * Need: * assert() */ #ifdef ZSTD_DEPS_NEED_ASSERT #ifndef ZSTD_DEPS_ASSERT #define ZSTD_DEPS_ASSERT -#include <assert.h> +#include <linux/kernel.h> + +#define assert(x) WARN_ON((x)) #endif /* ZSTD_DEPS_ASSERT */ #endif /* ZSTD_DEPS_NEED_ASSERT */ -/* Need: +/* + * This is only requested when DEBUGLEVEL >= 2, meaning + * it is disabled in production. + * Need: * ZSTD_DEBUG_PRINT() */ #ifdef ZSTD_DEPS_NEED_IO #ifndef ZSTD_DEPS_IO #define ZSTD_DEPS_IO -#include <stdio.h> -#define ZSTD_DEBUG_PRINT(...) fprintf(stderr, __VA_ARGS__) +#include <linux/printk.h> + +#define ZSTD_DEBUG_PRINT(...) pr_debug(__VA_ARGS__) #endif /* ZSTD_DEPS_IO */ #endif /* ZSTD_DEPS_NEED_IO */ -/* Only requested when <stdint.h> is known to be present. +/* + * Only requested when MSAN is enabled. * Need: * intptr_t */ @@ -142,7 +124,11 @@ static U64 ZSTD_div64(U64 dividend, U32 divisor) { #ifndef ZSTD_DEPS_STDINT #define ZSTD_DEPS_STDINT -#include <stdint.h> +/* + * The Linux Kernel doesn't provide intptr_t, only uintptr_t, which + * is an unsigned long. + */ +typedef long intptr_t #endif /* ZSTD_DEPS_STDINT */ #endif /* ZSTD_DEPS_NEED_STDINT */ diff --git a/lib/zstd/common/zstd_internal.h b/lib/zstd/common/zstd_internal.h index 8cbeedd858c4..14e0a8ff52a8 100644 --- a/lib/zstd/common/zstd_internal.h +++ b/lib/zstd/common/zstd_internal.h @@ -19,27 +19,18 @@ /*-************************************* * Dependencies ***************************************/ -#if !defined(ZSTD_NO_INTRINSICS) && defined(__aarch64__) -#include <arm_neon.h> -#endif #include "compiler.h" #include "mem.h" #include "debug.h" /* assert, DEBUGLOG, RAWLOG, g_debuglevel */ #include "error_private.h" #define ZSTD_STATIC_LINKING_ONLY -#include "../zstd.h" +#include <linux/zstd.h> #define FSE_STATIC_LINKING_ONLY #include "fse.h" #define HUF_STATIC_LINKING_ONLY #include "huf.h" -#ifndef XXH_STATIC_LINKING_ONLY -# define XXH_STATIC_LINKING_ONLY /* XXH64_state_t */ -#endif -#include "xxhash.h" /* XXH_reset, update, digest */ +#include <linux/xxhash.h> /* XXH_reset, update, digest */ -#if defined (__cplusplus) -extern "C" { -#endif /* ---- static assert (debug) --- */ #define ZSTD_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) @@ -228,20 +219,12 @@ static const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG; * Shared functions to include for inlining *********************************************/ static void ZSTD_copy8(void* dst, const void* src) { -#if !defined(ZSTD_NO_INTRINSICS) && defined(__aarch64__) - vst1_u8((uint8_t*)dst, vld1_u8((const uint8_t*)src)); -#else ZSTD_memcpy(dst, src, 8); -#endif } #define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; } static void ZSTD_copy16(void* dst, const void* src) { -#if !defined(ZSTD_NO_INTRINSICS) && defined(__aarch64__) - vst1q_u8((uint8_t*)dst, vld1q_u8((const uint8_t*)src)); -#else ZSTD_memcpy(dst, src, 16); -#endif } #define COPY16(d,s) { ZSTD_copy16(d,s); d+=16; s+=16; } @@ -395,17 +378,8 @@ MEM_STATIC U32 ZSTD_highbit32(U32 val) /* compress, dictBuilder, decodeCorpus { assert(val != 0); { -# if defined(_MSC_VER) /* Visual */ -# if STATIC_BMI2 == 1 - return _lzcnt_u32(val)^31; -# else - unsigned long r=0; - return _BitScanReverse(&r, val) ? (unsigned)r : 0; -# endif -# elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */ +# if (__GNUC__ >= 3) /* GCC Intrinsic */ return __builtin_clz (val) ^ 31; -# elif defined(__ICCARM__) /* IAR Intrinsic */ - return 31 - __CLZ(val); # else /* Software version */ static const U32 DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 }; U32 v = val; @@ -446,8 +420,5 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr, const void* src, size_t srcSize); -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_CCOMMON_H_MODULE */ diff --git a/lib/zstd/compress/fse_compress.c b/lib/zstd/compress/fse_compress.c index 304a82b3cc47..ce2fe1b23449 100644 --- a/lib/zstd/compress/fse_compress.c +++ b/lib/zstd/compress/fse_compress.c @@ -171,13 +171,6 @@ size_t FSE_buildCTable_wksp(FSE_CTable* ct, return 0; } -#ifndef ZSTD_NO_UNUSED_FUNCTIONS -size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog) -{ - FSE_FUNCTION_TYPE tableSymbol[FSE_MAX_TABLESIZE]; /* memset() is not necessary, even if static analyzer complain about it */ - return FSE_buildCTable_wksp(ct, normalizedCounter, maxSymbolValue, tableLog, tableSymbol, sizeof(tableSymbol)); -} -#endif @@ -628,78 +621,5 @@ size_t FSE_compress_usingCTable (void* dst, size_t dstSize, size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); } -#ifndef ZSTD_NO_UNUSED_FUNCTIONS -/* FSE_compress_wksp() : - * Same as FSE_compress2(), but using an externally allocated scratch buffer (`workSpace`). - * `wkspSize` size must be `(1<<tableLog)`. - */ -size_t FSE_compress_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize) -{ - BYTE* const ostart = (BYTE*) dst; - BYTE* op = ostart; - BYTE* const oend = ostart + dstSize; - - unsigned count[FSE_MAX_SYMBOL_VALUE+1]; - S16 norm[FSE_MAX_SYMBOL_VALUE+1]; - FSE_CTable* CTable = (FSE_CTable*)workSpace; - size_t const CTableSize = FSE_CTABLE_SIZE_U32(tableLog, maxSymbolValue); - void* scratchBuffer = (void*)(CTable + CTableSize); - size_t const scratchBufferSize = wkspSize - (CTableSize * sizeof(FSE_CTable)); - - /* init conditions */ - if (wkspSize < FSE_COMPRESS_WKSP_SIZE_U32(tableLog, maxSymbolValue)) return ERROR(tableLog_tooLarge); - if (srcSize <= 1) return 0; /* Not compressible */ - if (!maxSymbolValue) maxSymbolValue = FSE_MAX_SYMBOL_VALUE; - if (!tableLog) tableLog = FSE_DEFAULT_TABLELOG; - - /* Scan input and build symbol stats */ - { CHECK_V_F(maxCount, HIST_count_wksp(count, &maxSymbolValue, src, srcSize, scratchBuffer, scratchBufferSize) ); - if (maxCount == srcSize) return 1; /* only a single symbol in src : rle */ - if (maxCount == 1) return 0; /* each symbol present maximum once => not compressible */ - if (maxCount < (srcSize >> 7)) return 0; /* Heuristic : not compressible enough */ - } - - tableLog = FSE_optimalTableLog(tableLog, srcSize, maxSymbolValue); - CHECK_F( FSE_normalizeCount(norm, tableLog, count, srcSize, maxSymbolValue, /* useLowProbCount */ srcSize >= 2048) ); - - /* Write table description header */ - { CHECK_V_F(nc_err, FSE_writeNCount(op, oend-op, norm, maxSymbolValue, tableLog) ); - op += nc_err; - } - - /* Compress */ - CHECK_F( FSE_buildCTable_wksp(CTable, norm, maxSymbolValue, tableLog, scratchBuffer, scratchBufferSize) ); - { CHECK_V_F(cSize, FSE_compress_usingCTable(op, oend - op, src, srcSize, CTable) ); - if (cSize == 0) return 0; /* not enough space for compressed data */ - op += cSize; - } - - /* check compressibility */ - if ( (size_t)(op-ostart) >= srcSize-1 ) return 0; - - return op-ostart; -} - -typedef struct { - FSE_CTable CTable_max[FSE_CTABLE_SIZE_U32(FSE_MAX_TABLELOG, FSE_MAX_SYMBOL_VALUE)]; - union { - U32 hist_wksp[HIST_WKSP_SIZE_U32]; - BYTE scratchBuffer[1 << FSE_MAX_TABLELOG]; - } workspace; -} fseWkspMax_t; - -size_t FSE_compress2 (void* dst, size_t dstCapacity, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog) -{ - fseWkspMax_t scratchBuffer; - DEBUG_STATIC_ASSERT(sizeof(scratchBuffer) >= FSE_COMPRESS_WKSP_SIZE_U32(FSE_MAX_TABLELOG, FSE_MAX_SYMBOL_VALUE)); /* compilation failures here means scratchBuffer is not large enough */ - if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); - return FSE_compress_wksp(dst, dstCapacity, src, srcSize, maxSymbolValue, tableLog, &scratchBuffer, sizeof(scratchBuffer)); -} - -size_t FSE_compress (void* dst, size_t dstCapacity, const void* src, size_t srcSize) -{ - return FSE_compress2(dst, dstCapacity, src, srcSize, FSE_MAX_SYMBOL_VALUE, FSE_DEFAULT_TABLELOG); -} -#endif #endif /* FSE_COMMONDEFS_ONLY */ diff --git a/lib/zstd/compress/hist.c b/lib/zstd/compress/hist.c index a9659d11ad0c..83cb58d8c7b7 100644 --- a/lib/zstd/compress/hist.c +++ b/lib/zstd/compress/hist.c @@ -163,19 +163,3 @@ size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr, return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace, workSpaceSize); } -#ifndef ZSTD_NO_UNUSED_FUNCTIONS -/* fast variant (unsafe : won't check if src contains values beyond count[] limit) */ -size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr, - const void* source, size_t sourceSize) -{ - unsigned tmpCounters[HIST_WKSP_SIZE_U32]; - return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, tmpCounters, sizeof(tmpCounters)); -} - -size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr, - const void* src, size_t srcSize) -{ - unsigned tmpCounters[HIST_WKSP_SIZE_U32]; - return HIST_count_wksp(count, maxSymbolValuePtr, src, srcSize, tmpCounters, sizeof(tmpCounters)); -} -#endif diff --git a/lib/zstd/compress/huf_compress.c b/lib/zstd/compress/huf_compress.c index 67f0c928a556..9f77d05f4894 100644 --- a/lib/zstd/compress/huf_compress.c +++ b/lib/zstd/compress/huf_compress.c @@ -15,9 +15,6 @@ /* ************************************************************** * Compiler specifics ****************************************************************/ -#ifdef _MSC_VER /* Visual Studio */ -# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ -#endif /* ************************************************************** @@ -765,35 +762,3 @@ size_t HUF_compress4X_repeat (void* dst, size_t dstSize, hufTable, repeat, preferRepeat, bmi2); } -#ifndef ZSTD_NO_UNUSED_FUNCTIONS -/** HUF_buildCTable() : - * @return : maxNbBits - * Note : count is used before tree is written, so they can safely overlap - */ -size_t HUF_buildCTable (HUF_CElt* tree, const unsigned* count, unsigned maxSymbolValue, unsigned maxNbBits) -{ - HUF_buildCTable_wksp_tables workspace; - return HUF_buildCTable_wksp(tree, count, maxSymbolValue, maxNbBits, &workspace, sizeof(workspace)); -} - -size_t HUF_compress1X (void* dst, size_t dstSize, - const void* src, size_t srcSize, - unsigned maxSymbolValue, unsigned huffLog) -{ - unsigned workSpace[HUF_WORKSPACE_SIZE_U32]; - return HUF_compress1X_wksp(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, workSpace, sizeof(workSpace)); -} - -size_t HUF_compress2 (void* dst, size_t dstSize, - const void* src, size_t srcSize, - unsigned maxSymbolValue, unsigned huffLog) -{ - unsigned workSpace[HUF_WORKSPACE_SIZE_U32]; - return HUF_compress4X_wksp(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, workSpace, sizeof(workSpace)); -} - -size_t HUF_compress (void* dst, size_t maxDstSize, const void* src, size_t srcSize) -{ - return HUF_compress2(dst, maxDstSize, src, srcSize, 255, HUF_TABLELOG_DEFAULT); -} -#endif diff --git a/lib/zstd/compress/zstd_compress.c b/lib/zstd/compress/zstd_compress.c index 61369687a869..106e3b81a3f8 100644 --- a/lib/zstd/compress/zstd_compress.c +++ b/lib/zstd/compress/zstd_compress.c @@ -137,9 +137,6 @@ static void ZSTD_freeCCtxContent(ZSTD_CCtx* cctx) assert(cctx != NULL); assert(cctx->staticSize == 0); ZSTD_clearAllDicts(cctx); -#ifdef ZSTD_MULTITHREAD - ZSTDMT_freeCCtx(cctx->mtctx); cctx->mtctx = NULL; -#endif ZSTD_cwksp_free(&cctx->workspace, cctx->customMem); } @@ -161,12 +158,8 @@ size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx) static size_t ZSTD_sizeof_mtctx(const ZSTD_CCtx* cctx) { -#ifdef ZSTD_MULTITHREAD - return ZSTDMT_sizeof_CCtx(cctx->mtctx); -#else (void)cctx; return 0; -#endif } @@ -327,30 +320,17 @@ ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param) case ZSTD_c_nbWorkers: bounds.lowerBound = 0; -#ifdef ZSTD_MULTITHREAD - bounds.upperBound = ZSTDMT_NBWORKERS_MAX; -#else bounds.upperBound = 0; -#endif return bounds; case ZSTD_c_jobSize: bounds.lowerBound = 0; -#ifdef ZSTD_MULTITHREAD - bounds.upperBound = ZSTDMT_JOBSIZE_MAX; -#else bounds.upperBound = 0; -#endif return bounds; case ZSTD_c_overlapLog: -#ifdef ZSTD_MULTITHREAD - bounds.lowerBound = ZSTD_OVERLAPLOG_MIN; - bounds.upperBound = ZSTD_OVERLAPLOG_MAX; -#else bounds.lowerBound = 0; bounds.upperBound = 0; -#endif return bounds; case ZSTD_c_enableLongDistanceMatching: @@ -624,48 +604,20 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams, } case ZSTD_c_nbWorkers : -#ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; -#else - FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), ""); - CCtxParams->nbWorkers = value; - return CCtxParams->nbWorkers; -#endif case ZSTD_c_jobSize : -#ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; -#else - /* Adjust to the minimum non-default value. */ - if (value != 0 && value < ZSTDMT_JOBSIZE_MIN) - value = ZSTDMT_JOBSIZE_MIN; - FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), ""); - assert(value >= 0); - CCtxParams->jobSize = value; - return CCtxParams->jobSize; -#endif case ZSTD_c_overlapLog : -#ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; -#else - FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), ""); - CCtxParams->overlapLog = value; - return CCtxParams->overlapLog; -#endif case ZSTD_c_rsyncable : -#ifndef ZSTD_MULTITHREAD RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading"); return 0; -#else - FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(ZSTD_c_overlapLog, &value), ""); - CCtxParams->rsyncable = value; - return CCtxParams->rsyncable; -#endif case ZSTD_c_enableLongDistanceMatching : CCtxParams->ldmParams.enableLdm = (value!=0); @@ -767,33 +719,15 @@ size_t ZSTD_CCtxParams_getParameter( *value = CCtxParams->literalCompressionMode; break; case ZSTD_c_nbWorkers : -#ifndef ZSTD_MULTITHREAD assert(CCtxParams->nbWorkers == 0); -#endif *value = CCtxParams->nbWorkers; break; case ZSTD_c_jobSize : -#ifndef ZSTD_MULTITHREAD RETURN_ERROR(parameter_unsupported, "not compiled with multithreading"); -#else - assert(CCtxParams->jobSize <= INT_MAX); - *value = (int)CCtxParams->jobSize; - break; -#endif case ZSTD_c_overlapLog : -#ifndef ZSTD_MULTITHREAD RETURN_ERROR(parameter_unsupported, "not compiled with multithreading"); -#else - *value = CCtxParams->overlapLog; - break; -#endif case ZSTD_c_rsyncable : -#ifndef ZSTD_MULTITHREAD RETURN_ERROR(parameter_unsupported, "not compiled with multithreading"); -#else - *value = CCtxParams->rsyncable; - break; -#endif case ZSTD_c_enableLongDistanceMatching : *value = CCtxParams->ldmParams.enableLdm; break; @@ -1248,11 +1182,6 @@ size_t ZSTD_estimateCStreamSize(int compressionLevel) */ ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx) { -#ifdef ZSTD_MULTITHREAD - if (cctx->appliedParams.nbWorkers > 0) { - return ZSTDMT_getFrameProgression(cctx->mtctx); - } -#endif { ZSTD_frameProgression fp; size_t const buffered = (cctx->inBuff == NULL) ? 0 : cctx->inBuffPos - cctx->inToCompress; @@ -1272,11 +1201,6 @@ ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx) */ size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx) { -#ifdef ZSTD_MULTITHREAD - if (cctx->appliedParams.nbWorkers > 0) { - return ZSTDMT_toFlushNow(cctx->mtctx); - } -#endif (void)cctx; return 0; /* over-simplification; could also check if context is currently running in streaming mode, and in which case, report how many bytes are left to be flushed within output buffer */ } @@ -1520,7 +1444,7 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc, (unsigned)pledgedSrcSize, zc->appliedParams.fParams.contentSizeFlag); zc->blockSize = blockSize; - XXH64_reset(&zc->xxhState, 0); + xxh64_reset(&zc->xxhState, 0); zc->stage = ZSTDcs_init; zc->dictID = 0; @@ -2604,7 +2528,7 @@ static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx, DEBUGLOG(5, "ZSTD_compress_frameChunk (blockSize=%u)", (unsigned)blockSize); if (cctx->appliedParams.fParams.checksumFlag && srcSize) - XXH64_update(&cctx->xxhState, src, srcSize); + xxh64_update(&cctx->xxhState, src, srcSize); while (remaining) { ZSTD_matchState_t* const ms = &cctx->blockState.matchState; @@ -3209,7 +3133,7 @@ static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity) } if (cctx->appliedParams.fParams.checksumFlag) { - U32 const checksum = (U32) XXH64_digest(&cctx->xxhState); + U32 const checksum = (U32) xxh64_digest(&cctx->xxhState); RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for checksum"); DEBUGLOG(4, "ZSTD_writeEpilogue: write checksum : %08X", (unsigned)checksum); MEM_writeLE32(op, checksum); @@ -3954,12 +3878,6 @@ static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs, static size_t ZSTD_nextInputSizeHint_MTorST(const ZSTD_CCtx* cctx) { -#ifdef ZSTD_MULTITHREAD - if (cctx->appliedParams.nbWorkers >= 1) { - assert(cctx->mtctx != NULL); - return ZSTDMT_nextInputSizeHint(cctx->mtctx); - } -#endif return ZSTD_nextInputSizeHint(cctx); } @@ -3995,28 +3913,6 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx, &cctx->requestedParams, cctx->pledgedSrcSizePlusOne-1, 0 /*dictSize*/); -#ifdef ZSTD_MULTITHREAD - if ((cctx->pledgedSrcSizePlusOne-1) <= ZSTDMT_JOBSIZE_MIN) { - params.nbWorkers = 0; /* do not invoke multi-threading when src size is too small */ - } - if (params.nbWorkers > 0) { - /* mt context creation */ - if (cctx->mtctx == NULL) { - DEBUGLOG(4, "ZSTD_compressStream2: creating new mtctx for nbWorkers=%u", - params.nbWorkers); - cctx->mtctx = ZSTDMT_createCCtx_advanced((U32)params.nbWorkers, cctx->customMem); - RETURN_ERROR_IF(cctx->mtctx == NULL, memory_allocation, "NULL pointer!"); - } - /* mt compression */ - DEBUGLOG(4, "call ZSTDMT_initCStream_internal as nbWorkers=%u", params.nbWorkers); - FORWARD_IF_ERROR( ZSTDMT_initCStream_internal( - cctx->mtctx, - prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType, - cctx->cdict, params, cctx->pledgedSrcSizePlusOne-1) , ""); - cctx->streamStage = zcss_load; - cctx->appliedParams.nbWorkers = params.nbWorkers; - } else -#endif { FORWARD_IF_ERROR( ZSTD_resetCStream_internal(cctx, prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType, cctx->cdict, @@ -4027,31 +3923,6 @@ size_t ZSTD_compressStream2( ZSTD_CCtx* cctx, /* end of transparent initialization stage */ /* compression stage */ -#ifdef ZSTD_MULTITHREAD - if (cctx->appliedParams.nbWorkers > 0) { - int const forceMaxProgress = (endOp == ZSTD_e_flush || endOp == ZSTD_e_end); - size_t flushMin; - assert(forceMaxProgress || endOp == ZSTD_e_continue /* Protection for a new flush type */); - if (cctx->cParamsChanged) { - ZSTDMT_updateCParams_whileCompressing(cctx->mtctx, &cctx->requestedParams); - cctx->cParamsChanged = 0; - } - do { - flushMin = ZSTDMT_compressStream_generic(cctx->mtctx, output, input, endOp); - if ( ZSTD_isError(flushMin) - || (endOp == ZSTD_e_end && flushMin == 0) ) { /* compression completed */ - ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only); - } - FORWARD_IF_ERROR(flushMin, "ZSTDMT_compressStream_generic failed"); - } while (forceMaxProgress && flushMin != 0 && output->pos < output->size); - DEBUGLOG(5, "completed ZSTD_compressStream2 delegating to ZSTDMT_compressStream_generic"); - /* Either we don't require maximum forward progress, we've finished the - * flush, or we are out of output space. - */ - assert(!forceMaxProgress || flushMin == 0 || output->pos == output->size); - return flushMin; - } -#endif FORWARD_IF_ERROR( ZSTD_compressStream_generic(cctx, output, input, endOp) , ""); DEBUGLOG(5, "completed ZSTD_compressStream2"); return cctx->outBuffContentSize - cctx->outBuffFlushedSize; /* remaining to flush */ diff --git a/lib/zstd/compress/zstd_compress_internal.h b/lib/zstd/compress/zstd_compress_internal.h index 4760f6b08c6a..9e2a28274276 100644 --- a/lib/zstd/compress/zstd_compress_internal.h +++ b/lib/zstd/compress/zstd_compress_internal.h @@ -20,13 +20,7 @@ ***************************************/ #include "../common/zstd_internal.h" #include "zstd_cwksp.h" -#ifdef ZSTD_MULTITHREAD -# include "zstdmt_compress.h" -#endif -#if defined (__cplusplus) -extern "C" { -#endif /*-************************************* @@ -248,7 +242,7 @@ struct ZSTD_CCtx_s { unsigned long long pledgedSrcSizePlusOne; /* this way, 0 (default) == unknown */ unsigned long long consumedSrcSize; unsigned long long producedCSize; - XXH64_state_t xxhState; + struct xxh64_state xxhState; ZSTD_customMem customMem; size_t staticSize; SeqCollector seqCollector; @@ -282,9 +276,6 @@ struct ZSTD_CCtx_s { ZSTD_prefixDict prefixDict; /* single-usage dictionary */ /* Multi-threading */ -#ifdef ZSTD_MULTITHREAD - ZSTDMT_CCtx* mtctx; -#endif }; typedef enum { ZSTD_dtlm_fast, ZSTD_dtlm_full } ZSTD_dictTableLoadMethod_e; @@ -500,14 +491,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val) { if (MEM_isLittleEndian()) { if (MEM_64bits()) { -# if defined(_MSC_VER) && defined(_WIN64) -# if STATIC_BMI2 - return _tzcnt_u64(val) >> 3; -# else - unsigned long r = 0; - return _BitScanForward64( &r, (U64)val ) ? (unsigned)(r >> 3) : 0; -# endif -# elif defined(__GNUC__) && (__GNUC__ >= 4) +# if (__GNUC__ >= 4) return (__builtin_ctzll((U64)val) >> 3); # else static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, @@ -521,10 +505,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val) return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58]; # endif } else { /* 32 bits */ -# if defined(_MSC_VER) - unsigned long r=0; - return _BitScanForward( &r, (U32)val ) ? (unsigned)(r >> 3) : 0; -# elif defined(__GNUC__) && (__GNUC__ >= 3) +# if (__GNUC__ >= 3) return (__builtin_ctz((U32)val) >> 3); # else static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, @@ -536,14 +517,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val) } } else { /* Big Endian CPU */ if (MEM_64bits()) { -# if defined(_MSC_VER) && defined(_WIN64) -# if STATIC_BMI2 - return _lzcnt_u64(val) >> 3; -# else - unsigned long r = 0; - return _BitScanReverse64(&r, (U64)val) ? (unsigned)(r >> 3) : 0; -# endif -# elif defined(__GNUC__) && (__GNUC__ >= 4) +# if (__GNUC__ >= 4) return (__builtin_clzll(val) >> 3); # else unsigned r; @@ -554,10 +528,7 @@ static unsigned ZSTD_NbCommonBytes (size_t val) return r; # endif } else { /* 32 bits */ -# if defined(_MSC_VER) - unsigned long r = 0; - return _BitScanReverse( &r, (unsigned long)val ) ? (unsigned)(r >> 3) : 0; -# elif defined(__GNUC__) && (__GNUC__ >= 3) +# if (__GNUC__ >= 3) return (__builtin_clz((U32)val) >> 3); # else unsigned r; @@ -1041,9 +1012,6 @@ MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max) #endif -#if defined (__cplusplus) -} -#endif /* =============================================================== * Shared internal declarations diff --git a/lib/zstd/compress/zstd_compress_superblock.h b/lib/zstd/compress/zstd_compress_superblock.h index 07f4cb1dc646..f65da639589d 100644 --- a/lib/zstd/compress/zstd_compress_superblock.h +++ b/lib/zstd/compress/zstd_compress_superblock.h @@ -15,7 +15,7 @@ * Dependencies ***************************************/ -#include "../zstd.h" /* ZSTD_CCtx */ +#include <linux/zstd.h> /* ZSTD_CCtx */ /*-************************************* * Target Compressed Block Size diff --git a/lib/zstd/compress/zstd_cwksp.h b/lib/zstd/compress/zstd_cwksp.h index fdbc12ffe8fc..3e835d52bec3 100644 --- a/lib/zstd/compress/zstd_cwksp.h +++ b/lib/zstd/compress/zstd_cwksp.h @@ -16,9 +16,6 @@ ***************************************/ #include "../common/zstd_internal.h" -#if defined (__cplusplus) -extern "C" { -#endif /*-************************************* * Constants @@ -523,8 +520,5 @@ MEM_STATIC void ZSTD_cwksp_bump_oversized_duration( } } -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_CWKSP_H */ diff --git a/lib/zstd/compress/zstd_double_fast.h b/lib/zstd/compress/zstd_double_fast.h index 14d944d69bc1..f1b3882a11e9 100644 --- a/lib/zstd/compress/zstd_double_fast.h +++ b/lib/zstd/compress/zstd_double_fast.h @@ -11,9 +11,6 @@ #ifndef ZSTD_DOUBLE_FAST_H #define ZSTD_DOUBLE_FAST_H -#if defined (__cplusplus) -extern "C" { -#endif #include "../common/mem.h" /* U32 */ #include "zstd_compress_internal.h" /* ZSTD_CCtx, size_t */ @@ -31,8 +28,5 @@ size_t ZSTD_compressBlock_doubleFast_extDict( void const* src, size_t srcSize); -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_DOUBLE_FAST_H */ diff --git a/lib/zstd/compress/zstd_fast.h b/lib/zstd/compress/zstd_fast.h index cf6aaa8e6750..9487d0791308 100644 --- a/lib/zstd/compress/zstd_fast.h +++ b/lib/zstd/compress/zstd_fast.h @@ -11,9 +11,6 @@ #ifndef ZSTD_FAST_H #define ZSTD_FAST_H -#if defined (__cplusplus) -extern "C" { -#endif #include "../common/mem.h" /* U32 */ #include "zstd_compress_internal.h" @@ -30,8 +27,5 @@ size_t ZSTD_compressBlock_fast_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_FAST_H */ diff --git a/lib/zstd/compress/zstd_lazy.c b/lib/zstd/compress/zstd_lazy.c index 6371863f35d5..acd02c6d7cc7 100644 --- a/lib/zstd/compress/zstd_lazy.c +++ b/lib/zstd/compress/zstd_lazy.c @@ -681,7 +681,7 @@ ZSTD_compressBlock_lazy_generic( } /* Match Loop */ -#if defined(__GNUC__) && defined(__x86_64__) +#if defined(__x86_64__) /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the * code alignment is perturbed. To fix the instability align the loop on 32-bytes. */ @@ -958,7 +958,7 @@ size_t ZSTD_compressBlock_lazy_extDict_generic( ip += (ip == prefixStart); /* Match Loop */ -#if defined(__GNUC__) && defined(__x86_64__) +#if defined(__x86_64__) /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the * code alignment is perturbed. To fix the instability align the loop on 32-bytes. */ diff --git a/lib/zstd/compress/zstd_lazy.h b/lib/zstd/compress/zstd_lazy.h index 581936f03bd4..2596b587d82e 100644 --- a/lib/zstd/compress/zstd_lazy.h +++ b/lib/zstd/compress/zstd_lazy.h @@ -11,9 +11,6 @@ #ifndef ZSTD_LAZY_H #define ZSTD_LAZY_H -#if defined (__cplusplus) -extern "C" { -#endif #include "zstd_compress_internal.h" @@ -60,8 +57,5 @@ size_t ZSTD_compressBlock_btlazy2_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], void const* src, size_t srcSize); -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_LAZY_H */ diff --git a/lib/zstd/compress/zstd_ldm.h b/lib/zstd/compress/zstd_ldm.h index 229ea05a9e1e..1fd97077aee9 100644 --- a/lib/zstd/compress/zstd_ldm.h +++ b/lib/zstd/compress/zstd_ldm.h @@ -11,12 +11,9 @@ #ifndef ZSTD_LDM_H #define ZSTD_LDM_H -#if defined (__cplusplus) -extern "C" { -#endif #include "zstd_compress_internal.h" /* ldmParams_t, U32 */ -#include "../zstd.h" /* ZSTD_CCtx, size_t */ +#include <linux/zstd.h> /* ZSTD_CCtx, size_t */ /*-************************************* * Long distance matching @@ -103,8 +100,5 @@ size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize); void ZSTD_ldm_adjustParameters(ldmParams_t* params, ZSTD_compressionParameters const* cParams); -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_FAST_H */ diff --git a/lib/zstd/compress/zstd_opt.h b/lib/zstd/compress/zstd_opt.h index 9aba8a9018c5..7218323151eb 100644 --- a/lib/zstd/compress/zstd_opt.h +++ b/lib/zstd/compress/zstd_opt.h @@ -11,9 +11,6 @@ #ifndef ZSTD_OPT_H #define ZSTD_OPT_H -#if defined (__cplusplus) -extern "C" { -#endif #include "zstd_compress_internal.h" @@ -49,8 +46,5 @@ size_t ZSTD_compressBlock_btultra_extDict( * because btultra2 is not meant to work with dictionaries * and is only specific for the first block (no prefix) */ -#if defined (__cplusplus) -} -#endif #endif /* ZSTD_OPT_H */ diff --git a/lib/zstd/decompress/huf_decompress.c b/lib/zstd/decompress/huf_decompress.c index 141820671852..17976824b0d3 100644 --- a/lib/zstd/decompress/huf_decompress.c +++ b/lib/zstd/decompress/huf_decompress.c @@ -1203,148 +1203,3 @@ size_t HUF_decompress4X_hufOnly_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t ds } } -#ifndef ZSTD_NO_UNUSED_FUNCTIONS -#ifndef HUF_FORCE_DECOMPRESS_X2 -size_t HUF_readDTableX1(HUF_DTable* DTable, const void* src, size_t srcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_readDTableX1_wksp(DTable, src, srcSize, - workSpace, sizeof(workSpace)); -} - -size_t HUF_decompress1X1_DCtx(HUF_DTable* DCtx, void* dst, size_t dstSize, - const void* cSrc, size_t cSrcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_decompress1X1_DCtx_wksp(DCtx, dst, dstSize, cSrc, cSrcSize, - workSpace, sizeof(workSpace)); -} - -size_t HUF_decompress1X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ - HUF_CREATE_STATIC_DTABLEX1(DTable, HUF_TABLELOG_MAX); - return HUF_decompress1X1_DCtx (DTable, dst, dstSize, cSrc, cSrcSize); -} -#endif - -#ifndef HUF_FORCE_DECOMPRESS_X1 -size_t HUF_readDTableX2(HUF_DTable* DTable, const void* src, size_t srcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_readDTableX2_wksp(DTable, src, srcSize, - workSpace, sizeof(workSpace)); -} - -size_t HUF_decompress1X2_DCtx(HUF_DTable* DCtx, void* dst, size_t dstSize, - const void* cSrc, size_t cSrcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_decompress1X2_DCtx_wksp(DCtx, dst, dstSize, cSrc, cSrcSize, - workSpace, sizeof(workSpace)); -} - -size_t HUF_decompress1X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ - HUF_CREATE_STATIC_DTABLEX2(DTable, HUF_TABLELOG_MAX); - return HUF_decompress1X2_DCtx(DTable, dst, dstSize, cSrc, cSrcSize); -} -#endif - -#ifndef HUF_FORCE_DECOMPRESS_X2 -size_t HUF_decompress4X1_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, - workSpace, sizeof(workSpace)); -} -size_t HUF_decompress4X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ - HUF_CREATE_STATIC_DTABLEX1(DTable, HUF_TABLELOG_MAX); - return HUF_decompress4X1_DCtx(DTable, dst, dstSize, cSrc, cSrcSize); -} -#endif - -#ifndef HUF_FORCE_DECOMPRESS_X1 -size_t HUF_decompress4X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, - const void* cSrc, size_t cSrcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, - workSpace, sizeof(workSpace)); -} - -size_t HUF_decompress4X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ - HUF_CREATE_STATIC_DTABLEX2(DTable, HUF_TABLELOG_MAX); - return HUF_decompress4X2_DCtx(DTable, dst, dstSize, cSrc, cSrcSize); -} -#endif - -typedef size_t (*decompressionAlgo)(void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); - -size_t HUF_decompress (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ -#if !defined(HUF_FORCE_DECOMPRESS_X1) && !defined(HUF_FORCE_DECOMPRESS_X2) - static const decompressionAlgo decompress[2] = { HUF_decompress4X1, HUF_decompress4X2 }; -#endif - - /* validation checks */ - if (dstSize == 0) return ERROR(dstSize_tooSmall); - if (cSrcSize > dstSize) return ERROR(corruption_detected); /* invalid */ - if (cSrcSize == dstSize) { ZSTD_memcpy(dst, cSrc, dstSize); return dstSize; } /* not compressed */ - if (cSrcSize == 1) { ZSTD_memset(dst, *(const BYTE*)cSrc, dstSize); return dstSize; } /* RLE */ - - { U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize); -#if defined(HUF_FORCE_DECOMPRESS_X1) - (void)algoNb; - assert(algoNb == 0); - return HUF_decompress4X1(dst, dstSize, cSrc, cSrcSize); -#elif defined(HUF_FORCE_DECOMPRESS_X2) - (void)algoNb; - assert(algoNb == 1); - return HUF_decompress4X2(dst, dstSize, cSrc, cSrcSize); -#else - return decompress[algoNb](dst, dstSize, cSrc, cSrcSize); -#endif - } -} - -size_t HUF_decompress4X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ - /* validation checks */ - if (dstSize == 0) return ERROR(dstSize_tooSmall); - if (cSrcSize > dstSize) return ERROR(corruption_detected); /* invalid */ - if (cSrcSize == dstSize) { ZSTD_memcpy(dst, cSrc, dstSize); return dstSize; } /* not compressed */ - if (cSrcSize == 1) { ZSTD_memset(dst, *(const BYTE*)c Src, dstSize); return dstSize; } /* RLE */ - - { U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize); -#if defined(HUF_FORCE_DECOMPRESS_X1) - (void)algoNb; - assert(algoNb == 0); - return HUF_decompress4X1_DCtx(dctx, dst, dstSize, cSrc, cSrcSize); -#elif defined(HUF_FORCE_DECOMPRESS_X2) - (void)algoNb; - assert(algoNb == 1); - return HUF_decompress4X2_DCtx(dctx, dst, dstSize, cSrc, cSrcSize); -#else - return algoNb ? HUF_decompress4X2_DCtx(dctx, dst, dstSize, cSrc, cSrcSize) : - HUF_decompress4X1_DCtx(dctx, dst, dstSize, cSrc, cSrcSize) ; -#endif - } -} - -size_t HUF_decompress4X_hufOnly(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_decompress4X_hufOnly_wksp(dctx, dst, dstSize, cSrc, cSrcSize, - workSpace, sizeof(workSpace)); -} - -size_t HUF_decompress1X_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, - const void* cSrc, size_t cSrcSize) -{ - U32 workSpace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; - return HUF_decompress1X_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, - workSpace, sizeof(workSpace)); -} -#endif diff --git a/lib/zstd/decompress/zstd_ddict.c b/lib/zstd/decompress/zstd_ddict.c index da7d3f82d28d..75c1eac3894d 100644 --- a/lib/zstd/decompress/zstd_ddict.c +++ b/lib/zstd/decompress/zstd_ddict.c @@ -24,9 +24,6 @@ #include "zstd_decompress_internal.h" #include "zstd_ddict.h" -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) -# include "../legacy/zstd_legacy.h" -#endif diff --git a/lib/zstd/decompress/zstd_ddict.h b/lib/zstd/decompress/zstd_ddict.h index 8906a71c9404..1da6e5bd86f9 100644 --- a/lib/zstd/decompress/zstd_ddict.h +++ b/lib/zstd/decompress/zstd_ddict.h @@ -16,7 +16,7 @@ * Dependencies *********************************************************/ #include "../common/zstd_deps.h" /* size_t */ -#include "../zstd.h" /* ZSTD_DDict, and several public functions */ +#include <linux/zstd.h> /* ZSTD_DDict, and several public functions */ /*-******************************************************* diff --git a/lib/zstd/decompress/zstd_decompress.c b/lib/zstd/decompress/zstd_decompress.c index 3deae1d5e6f4..5a1a42268dda 100644 --- a/lib/zstd/decompress/zstd_decompress.c +++ b/lib/zstd/decompress/zstd_decompress.c @@ -26,9 +26,6 @@ * LEGACY_SUPPORT : * if set to 1+, ZSTD_decompress() can decode older formats (v0.1+) */ -#ifndef ZSTD_LEGACY_SUPPORT -# define ZSTD_LEGACY_SUPPORT 0 -#endif /*! * MAXWINDOWSIZE_DEFAULT : @@ -67,9 +64,6 @@ #include "zstd_ddict.h" /* ZSTD_DDictDictContent */ #include "zstd_decompress_block.h" /* ZSTD_decompressBlock_internal */ -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) -# include "../legacy/zstd_legacy.h" -#endif /*-************************************************************* @@ -168,10 +162,6 @@ size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx) ZSTD_clearDict(dctx); ZSTD_customFree(dctx->inBuff, cMem); dctx->inBuff = NULL; -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) - if (dctx->legacyContext) - ZSTD_freeLegacyStreamContext(dctx->legacyContext, dctx->previousLegacyVersion); -#endif ZSTD_customFree(dctx, cMem); return 0; } @@ -201,9 +191,6 @@ unsigned ZSTD_isFrame(const void* buffer, size_t size) if (magic == ZSTD_MAGICNUMBER) return 1; if ((magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) return 1; } -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) - if (ZSTD_isLegacy(buffer, size)) return 1; -#endif return 0; } @@ -338,12 +325,6 @@ size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t src * - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) */ unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize) { -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) - if (ZSTD_isLegacy(src, srcSize)) { - unsigned long long const ret = ZSTD_getDecompressedSize_legacy(src, srcSize); - return ret == 0 ? ZSTD_CONTENTSIZE_UNKNOWN : ret; - } -#endif { ZSTD_frameHeader zfh; if (ZSTD_getFrameHeader(&zfh, src, srcSize) != 0) return ZSTD_CONTENTSIZE_ERROR; @@ -449,7 +430,7 @@ static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx* dctx, const void* src, size_t he dictionary_wrong, ""); #endif dctx->validateChecksum = (dctx->fParams.checksumFlag && !dctx->forceIgnoreChecksum) ? 1 : 0; - if (dctx->validateChecksum) XXH64_reset(&dctx->xxhState, 0); + if (dctx->validateChecksum) xxh64_reset(&dctx->xxhState, 0); return 0; } @@ -466,10 +447,6 @@ static ZSTD_frameSizeInfo ZSTD_findFrameSizeInfo(const void* src, size_t srcSize ZSTD_frameSizeInfo frameSizeInfo; ZSTD_memset(&frameSizeInfo, 0, sizeof(ZSTD_frameSizeInfo)); -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) - if (ZSTD_isLegacy(src, srcSize)) - return ZSTD_findFrameSizeInfoLegacy(src, srcSize); -#endif if ((srcSize >= ZSTD_SKIPPABLEHEADERSIZE) && (MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { @@ -665,7 +642,7 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx, if (ZSTD_isError(decodedSize)) return decodedSize; if (dctx->validateChecksum) - XXH64_update(&dctx->xxhState, op, decodedSize); + xxh64_update(&dctx->xxhState, op, decodedSize); if (decodedSize != 0) op += decodedSize; assert(ip != NULL); @@ -681,7 +658,7 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx, if (dctx->fParams.checksumFlag) { /* Frame content checksum verification */ RETURN_ERROR_IF(remainingSrcSize<4, checksum_wrong, ""); if (!dctx->forceIgnoreChecksum) { - U32 const checkCalc = (U32)XXH64_digest(&dctx->xxhState); + U32 const checkCalc = (U32)xxh64_digest(&dctx->xxhState); U32 checkRead; checkRead = MEM_readLE32(ip); RETURN_ERROR_IF(checkRead != checkCalc, checksum_wrong, ""); @@ -715,27 +692,6 @@ static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx, while (srcSize >= ZSTD_startingInputLength(dctx->format)) { -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT >= 1) - if (ZSTD_isLegacy(src, srcSize)) { - size_t decodedSize; - size_t const frameSize = ZSTD_findFrameCompressedSizeLegacy(src, srcSize); - if (ZSTD_isError(frameSize)) return frameSize; - RETURN_ERROR_IF(dctx->staticSize, memory_allocation, - "legacy support is not compatible with static dctx"); - - decodedSize = ZSTD_decompressLegacy(dst, dstCapacity, src, frameSize, dict, dictSize); - if (ZSTD_isError(decodedSize)) return decodedSize; - - assert(decodedSize <=- dstCapacity); - dst = (BYTE*)dst + decodedSize; - dstCapacity -= decodedSize; - - src = (const BYTE*)src + frameSize; - srcSize -= frameSize; - - continue; - } -#endif { U32 const magicNumber = MEM_readLE32(src); DEBUGLOG(4, "reading magic number %08X (expecting %08X)", @@ -982,7 +938,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c RETURN_ERROR_IF(rSize > dctx->fParams.blockSizeMax, corruption_detected, "Decompressed Block Size Exceeds Maximum"); DEBUGLOG(5, "ZSTD_decompressContinue: decoded size from block : %u", (unsigned)rSize); dctx->decodedSize += rSize; - if (dctx->validateChecksum) XXH64_update(&dctx->xxhState, dst, rSize); + if (dctx->validateChecksum) xxh64_update(&dctx->xxhState, dst, rSize); dctx->previousDstEnd = (char*)dst + rSize; /* Stay on the same stage until we are finished streaming the block. */ @@ -1014,7 +970,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c assert(srcSize == 4); /* guaranteed by dctx->expected */ { if (dctx->validateChecksum) { - U32 const h32 = (U32)XXH64_digest(&dctx->xxhState); + U32 const h32 = (U32)xxh64_digest(&dctx->xxhState); U32 const check32 = MEM_readLE32(src); DEBUGLOG(4, "ZSTD_decompressContinue: checksum : calculated %08X :: %08X read", (unsigned)h32, (unsigned)check32); RETURN_ERROR_IF(check32 != h32, checksum_wrong, ""); @@ -1647,36 +1603,9 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB case zdss_loadHeader : DEBUGLOG(5, "stage zdss_loadHeader (srcSize : %u)", (U32)(iend - ip)); -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) - if (zds->legacyVersion) { - RETURN_ERROR_IF(zds->staticSize, memory_allocation, - "legacy support is incompatible with static dctx"); - { size_t const hint = ZSTD_decompressLegacyStream(zds->legacyContext, zds->legacyVersion, output, input); - if (hint==0) zds->streamStage = zdss_init; - return hint; - } } -#endif { size_t const hSize = ZSTD_getFrameHeader_advanced(&zds->fParams, zds->headerBuffer, zds->lhSize, zds->format); DEBUGLOG(5, "header size : %u", (U32)hSize); if (ZSTD_isError(hSize)) { -#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>=1) - U32 const legacyVersion = ZSTD_isLegacy(istart, iend-istart); - if (legacyVersion) { - ZSTD_DDict const* const ddict = ZSTD_getDDict(zds); - const void* const dict = ddict ? ZSTD_DDict_dictContent(ddict) : NULL; - size_t const dictSize = ddict ? ZSTD_DDict_dictSize(ddict) : 0; - DEBUGLOG(5, "ZSTD_decompressStream: detected legacy version v0.%u", legacyVersion); - RETURN_ERROR_IF(zds->staticSize, memory_allocation, - "legacy support is incompatible with static dctx"); - FORWARD_IF_ERROR(ZSTD_initLegacyStream(&zds->legacyContext, - zds->previousLegacyVersion, legacyVersion, - dict, dictSize), ""); - zds->legacyVersion = zds->previousLegacyVersion = legacyVersion; - { size_t const hint = ZSTD_decompressLegacyStream(zds->legacyContext, legacyVersion, output, input); - if (hint==0) zds->streamStage = zdss_init; /* or stay in stage zdss_loadHeader */ - return hint; - } } -#endif return hSize; /* error */ } if (hSize != 0) { /* need more input */ diff --git a/lib/zstd/decompress/zstd_decompress_block.c b/lib/zstd/decompress/zstd_decompress_block.c index bec82e857094..8627f839df3d 100644 --- a/lib/zstd/decompress/zstd_decompress_block.c +++ b/lib/zstd/decompress/zstd_decompress_block.c @@ -1030,7 +1030,7 @@ ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets, c * measure that it is worse, please put up a pull request. */ { -#if defined(__GNUC__) && !defined(__clang__) +#if !defined(__clang__) const int kUseUpdateFseState = 1; #else const int kUseUpdateFseState = 0; @@ -1138,7 +1138,7 @@ ZSTD_decompressSequences_body( ZSTD_DCtx* dctx, BIT_DStream_endOfBuffer < BIT_DStream_completed && BIT_DStream_completed < BIT_DStream_overflow); -#if defined(__GNUC__) && defined(__x86_64__) +#if defined(__x86_64__) /* Align the decompression loop to 32 + 16 bytes. * * zstd compiled with gcc-9 on an Intel i9-9900k shows 10% decompression diff --git a/lib/zstd/decompress/zstd_decompress_block.h b/lib/zstd/decompress/zstd_decompress_block.h index b5715c168e2b..15d801f19a89 100644 --- a/lib/zstd/decompress/zstd_decompress_block.h +++ b/lib/zstd/decompress/zstd_decompress_block.h @@ -16,7 +16,7 @@ * Dependencies *********************************************************/ #include "../common/zstd_deps.h" /* size_t */ -#include "../zstd.h" /* DCtx, and some public functions */ +#include <linux/zstd.h> /* DCtx, and some public functions */ #include "../common/zstd_internal.h" /* blockProperties_t, and some public functions */ #include "zstd_decompress_internal.h" /* ZSTD_seqSymbol */ diff --git a/lib/zstd/decompress/zstd_decompress_internal.h b/lib/zstd/decompress/zstd_decompress_internal.h index 82bf7c758de1..ed941b277a5d 100644 --- a/lib/zstd/decompress/zstd_decompress_internal.h +++ b/lib/zstd/decompress/zstd_decompress_internal.h @@ -123,7 +123,7 @@ struct ZSTD_DCtx_s ZSTD_dStage stage; U32 litEntropy; U32 fseEntropy; - XXH64_state_t xxhState; + struct xxh64_state xxhState; size_t headerSize; ZSTD_format_e format; ZSTD_forceIgnoreChecksum_e forceIgnoreChecksum; /* User specified: if == 1, will ignore checksums in compressed frame. Default == 0 */
From: Nick Terrell <terrelln@fb.com> This patchset upgrades the zstd library to the latest upstream release. The current zstd version in the kernel is a modified version of upstream zstd-1.3.1. At the time it was integrated, zstd wasn't ready to be used in the kernel as-is. But, it is now possible to use upstream zstd directly in the kernel. I have not yet release zstd-1.4.6 upstream. I want the zstd version in the kernel to match up with a known upstream release, so we know exactly what code is running. Whenever this patchset is ready for merge, I will cut a release at the upstream commit that gets merged. This should not be necessary for future releases. The kernel zstd library is automatically generated from upstream zstd. A script makes the necessary changes and imports it into the kernel. The changes are: 1. Replace all libc dependencies with kernel replacements and rewrite includes. 2. Remove unncessary portability macros like: #if defined(_MSC_VER). 3. Use the kernel xxhash instead of bundling it. This automation gets tested every commit by upstream's continuous integration. When we cut a new zstd release, we will submit a patch to the kernel to update the zstd version in the kernel. I've updated zstd to upstream with one big patch because every commit must build, so that precludes partial updates. Since the commit is 100% generated, I hope the review burden is lightened. I considered replaying upstream commits, but that is not possible because there have been ~3500 upstream commits since the last zstd import, and the commits don't all build individually. The bulk update preserves bisectablity because bugs can be bisected to the zstd version update. At that point the update can be reverted, and we can work with upstream to find and fix the bug. After this big switch in how the kernel consumes zstd, future patches will be smaller, because they will only have one upstream release worth of changes each. This patchset comes in 3 parts: 1. The first 2 patches prepare for the zstd upgrade. The first patch adds a compatibility wrapper so zstd can be upgraded without modifying any callers. The second patch adds an indirection for the lib/decompress_unzstd.c including of all decompression source files. 2. Import zstd-1.4.6. This patch is completely generated from upstream using automated tooling. 3. Update all callers to the zstd-1.4.6 API then delete the compatibility wrapper. I tested every caller of zstd on x86_64. I tested both after the 1.4.6 upgrade using the compatibility wrapper, and after the final patch in this series. I had problems with F2FS, where I had file truncation both before and after this series, so I would appreciate help testing it. All other callers were good. I tested kernel and initramfs decompression in i386 and arm. I ran benchmarks to compare the current zstd in the kernel with zstd-1.4.6. I benchmarked on x86_64 using QEMU with KVM enabled on an Intel i9-9900k. I found: * BtrFS zstd compression at levels 1 and 3 is 5% faster * BtrFS zstd decompression+read is 15% faster * SquashFS zstd decompression+read is 15% faster * ZRAM decompression+read is 30% faster * Kernel zstd decompression is 35% faster * Initramfs zstd decompression+build is 5% faster The latest zstd also offers bug fixes and a 1 KB reduction in stack uage during compression. Please let me know if there is anything that I can do to ease the way for these patches. I think it is important because it gets large performance improvements, contains bug fixes, and is switching to a more maintainable model of consuming upstream zstd directly, making it easy to keep up to date. Best, Nick Terrell Nick Terrell (9): lib: zstd: Add zstd compatibility wrapper lib: zstd: Add decompress_sources.h for decompress_unzstd lib: zstd: Upgrade to latest upstream zstd version 1.4.6 crypto: zstd: Switch to zstd-1.4.6 API btrfs: zstd: Switch to the zstd-1.4.6 API f2fs: zstd: Switch to the zstd-1.4.6 API squashfs: zstd: Switch to the zstd-1.4.6 API lib: unzstd: Switch to the zstd-1.4.6 API lib: zstd: Remove zstd compatibility wrapper crypto/zstd.c | 22 +- fs/btrfs/zstd.c | 46 +- fs/f2fs/compress.c | 100 +- fs/squashfs/zstd_wrapper.c | 7 +- include/linux/zstd.h | 3019 ++++++++---- include/linux/zstd_errors.h | 76 + lib/decompress_unzstd.c | 44 +- lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 379 -- lib/zstd/common/bitstream.h | 437 ++ lib/zstd/common/compiler.h | 134 + lib/zstd/common/cpu.h | 194 + lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 101 + lib/zstd/common/entropy_common.c | 355 ++ lib/zstd/common/error_private.c | 55 + lib/zstd/common/error_private.h | 66 + lib/zstd/common/fse.h | 709 +++ lib/zstd/common/fse_decompress.c | 380 ++ lib/zstd/common/huf.h | 352 ++ lib/zstd/common/mem.h | 347 ++ lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 134 + lib/zstd/common/zstd_internal.h | 424 ++ lib/zstd/compress.c | 3485 -------------- lib/zstd/compress/fse_compress.c | 625 +++ lib/zstd/compress/hist.c | 165 + lib/zstd/compress/hist.h | 75 + lib/zstd/compress/huf_compress.c | 764 +++ lib/zstd/compress/zstd_compress.c | 4160 +++++++++++++++++ lib/zstd/compress/zstd_compress_internal.h | 1103 +++++ lib/zstd/compress/zstd_compress_literals.c | 158 + lib/zstd/compress/zstd_compress_literals.h | 29 + lib/zstd/compress/zstd_compress_sequences.c | 433 ++ lib/zstd/compress/zstd_compress_sequences.h | 54 + lib/zstd/compress/zstd_compress_superblock.c | 849 ++++ lib/zstd/compress/zstd_compress_superblock.h | 32 + lib/zstd/compress/zstd_cwksp.h | 524 +++ lib/zstd/compress/zstd_double_fast.c | 521 +++ lib/zstd/compress/zstd_double_fast.h | 32 + lib/zstd/compress/zstd_fast.c | 496 ++ lib/zstd/compress/zstd_fast.h | 31 + lib/zstd/compress/zstd_lazy.c | 1138 +++++ lib/zstd/compress/zstd_lazy.h | 61 + lib/zstd/compress/zstd_ldm.c | 619 +++ lib/zstd/compress/zstd_ldm.h | 104 + lib/zstd/compress/zstd_opt.c | 1200 +++++ lib/zstd/compress/zstd_opt.h | 50 + lib/zstd/decompress.c | 2531 ---------- lib/zstd/decompress/huf_decompress.c | 1205 +++++ lib/zstd/decompress/zstd_ddict.c | 241 + lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 1836 ++++++++ lib/zstd/decompress/zstd_decompress_block.c | 1540 ++++++ lib/zstd/decompress/zstd_decompress_block.h | 62 + .../decompress/zstd_decompress_internal.h | 195 + lib/zstd/decompress_sources.h | 18 + lib/zstd/entropy_common.c | 243 - lib/zstd/error_private.h | 53 - lib/zstd/fse.h | 575 --- lib/zstd/fse_compress.c | 795 ---- lib/zstd/fse_decompress.c | 325 -- lib/zstd/huf.h | 212 - lib/zstd/huf_compress.c | 772 --- lib/zstd/huf_decompress.c | 960 ---- lib/zstd/mem.h | 151 - lib/zstd/zstd_common.c | 75 - lib/zstd/zstd_compress_module.c | 79 + lib/zstd/zstd_decompress_module.c | 79 + lib/zstd/zstd_internal.h | 273 -- lib/zstd/zstd_opt.h | 1014 ---- 71 files changed, 24497 insertions(+), 13012 deletions(-) create mode 100644 include/linux/zstd_errors.h delete mode 100644 lib/zstd/bitstream.h create mode 100644 lib/zstd/common/bitstream.h create mode 100644 lib/zstd/common/compiler.h create mode 100644 lib/zstd/common/cpu.h create mode 100644 lib/zstd/common/debug.c create mode 100644 lib/zstd/common/debug.h create mode 100644 lib/zstd/common/entropy_common.c create mode 100644 lib/zstd/common/error_private.c create mode 100644 lib/zstd/common/error_private.h create mode 100644 lib/zstd/common/fse.h create mode 100644 lib/zstd/common/fse_decompress.c create mode 100644 lib/zstd/common/huf.h create mode 100644 lib/zstd/common/mem.h create mode 100644 lib/zstd/common/zstd_common.c create mode 100644 lib/zstd/common/zstd_deps.h create mode 100644 lib/zstd/common/zstd_internal.h delete mode 100644 lib/zstd/compress.c create mode 100644 lib/zstd/compress/fse_compress.c create mode 100644 lib/zstd/compress/hist.c create mode 100644 lib/zstd/compress/hist.h create mode 100644 lib/zstd/compress/huf_compress.c create mode 100644 lib/zstd/compress/zstd_compress.c create mode 100644 lib/zstd/compress/zstd_compress_internal.h create mode 100644 lib/zstd/compress/zstd_compress_literals.c create mode 100644 lib/zstd/compress/zstd_compress_literals.h create mode 100644 lib/zstd/compress/zstd_compress_sequences.c create mode 100644 lib/zstd/compress/zstd_compress_sequences.h create mode 100644 lib/zstd/compress/zstd_compress_superblock.c create mode 100644 lib/zstd/compress/zstd_compress_superblock.h create mode 100644 lib/zstd/compress/zstd_cwksp.h create mode 100644 lib/zstd/compress/zstd_double_fast.c create mode 100644 lib/zstd/compress/zstd_double_fast.h create mode 100644 lib/zstd/compress/zstd_fast.c create mode 100644 lib/zstd/compress/zstd_fast.h create mode 100644 lib/zstd/compress/zstd_lazy.c create mode 100644 lib/zstd/compress/zstd_lazy.h create mode 100644 lib/zstd/compress/zstd_ldm.c create mode 100644 lib/zstd/compress/zstd_ldm.h create mode 100644 lib/zstd/compress/zstd_opt.c create mode 100644 lib/zstd/compress/zstd_opt.h delete mode 100644 lib/zstd/decompress.c create mode 100644 lib/zstd/decompress/huf_decompress.c create mode 100644 lib/zstd/decompress/zstd_ddict.c create mode 100644 lib/zstd/decompress/zstd_ddict.h create mode 100644 lib/zstd/decompress/zstd_decompress.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.h create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h create mode 100644 lib/zstd/decompress_sources.h delete mode 100644 lib/zstd/entropy_common.c delete mode 100644 lib/zstd/error_private.h delete mode 100644 lib/zstd/fse.h delete mode 100644 lib/zstd/fse_compress.c delete mode 100644 lib/zstd/fse_decompress.c delete mode 100644 lib/zstd/huf.h delete mode 100644 lib/zstd/huf_compress.c delete mode 100644 lib/zstd/huf_decompress.c delete mode 100644 lib/zstd/mem.h delete mode 100644 lib/zstd/zstd_common.c create mode 100644 lib/zstd/zstd_compress_module.c create mode 100644 lib/zstd/zstd_decompress_module.c delete mode 100644 lib/zstd/zstd_internal.h delete mode 100644 lib/zstd/zstd_opt.h