Message ID | 20240603064002.266116-1-tao1.su@linux.intel.com (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Series | [v2] KVM: x86: Advertise AVX10.1 CPUID to userspace | expand |
On Mon, Jun 03, 2024 at 02:40:02PM +0800, Tao Su wrote: > Advertise AVX10.1 related CPUIDs, i.e. report AVX10 support bit via > CPUID.(EAX=07H, ECX=01H):EDX[bit 19] and new CPUID leaf 0x24H so that > guest OS and applications can query the AVX10.1 CPUIDs directly. Intel > AVX10 represents the first major new vector ISA since the introduction of > Intel AVX512, which will establish a common, converged vector instruction > set across all Intel architectures[1]. > > AVX10.1 is an early version of AVX10, that enumerates the Intel AVX512 > instruction set at 128, 256, and 512 bits which is enabled on > Granite Rapids. I.e., AVX10.1 is only a new CPUID enumeration with no > VMX capability, Embedded rounding and Suppress All Exceptions (SAE), > which will be introduced in AVX10.2. > > Advertising AVX10.1 is safe because kernel doesn't enable AVX10.1 which is > on KVM-only leaf now, just the CPUID checking is changed when using AVX512 > related instructions, e.g. if using one AVX512 instruction needs to check > (AVX512 AND AVX512DQ), it can check ((AVX512 AND AVX512DQ) OR AVX10.1) > after checking XCR0[7:5]. > > The versions of AVX10 are expected to be inclusive, e.g. version N+1 is > a superset of version N. Per the spec, the version can never be 0, just > advertise AVX10.1 if it's supported in hardware. > > As more and more AVX related CPUIDs are added (it would have resulted in > around 40-50 CPUID flags when developing AVX10), the versioning approach > is introduced. But incrementing version numbers are bad for virtualization. > E.g. if AVX10.2 has a feature that shouldn't be enumerated to guests for > whatever reason, then KVM can't enumerate any "later" features either, > because the only way to hide the problematic AVX10.2 feature is to set the > version to AVX10.1 or lower[2]. But most AVX features are just passed > through and don’t have virtualization controls, so AVX10 should not be > problematic in practice. > > [1] https://cdrdv2.intel.com/v1/dl/getContent/784267 > [2] https://lore.kernel.org/all/Zkz5Ak0PQlAN8DxK@google.com/ > Hi Sean, do you have any suggestions for this patch? Thanks!
On Mon, Jun 03, 2024, Tao Su wrote: > Advertise AVX10.1 related CPUIDs, i.e. report AVX10 support bit via > CPUID.(EAX=07H, ECX=01H):EDX[bit 19] and new CPUID leaf 0x24H so that > guest OS and applications can query the AVX10.1 CPUIDs directly. Intel > AVX10 represents the first major new vector ISA since the introduction of > Intel AVX512, which will establish a common, converged vector instruction > set across all Intel architectures[1]. > > AVX10.1 is an early version of AVX10, that enumerates the Intel AVX512 > instruction set at 128, 256, and 512 bits which is enabled on > Granite Rapids. I.e., AVX10.1 is only a new CPUID enumeration with no > VMX capability, Embedded rounding and Suppress All Exceptions (SAE), > which will be introduced in AVX10.2. > > Advertising AVX10.1 is safe because kernel doesn't enable AVX10.1 which is I thought there is nothing to enable for AVX10.1? I.e. it's purely a new way to enumerate support, thus there will never be anything for the kernel to enable. > on KVM-only leaf now, just the CPUID checking is changed when using AVX512 > related instructions, e.g. if using one AVX512 instruction needs to check > (AVX512 AND AVX512DQ), it can check ((AVX512 AND AVX512DQ) OR AVX10.1) > after checking XCR0[7:5]. > > The versions of AVX10 are expected to be inclusive, e.g. version N+1 is > a superset of version N. Per the spec, the version can never be 0, just > advertise AVX10.1 if it's supported in hardware. I think it's also worth calling out that advertising AVX10_{128,256,512} needs to land in the same patch (this patch) as AVX10 (and thus AVX10.1), because otherwise KVM would advertise an impossible CPU model, e.g. with AVX512 but not AVX10.1/512, which per "Feature Differences Between Intel® AVX-512 and Intel® AVX10" should be impossible. > As more and more AVX related CPUIDs are added (it would have resulted in > around 40-50 CPUID flags when developing AVX10), the versioning approach > is introduced. But incrementing version numbers are bad for virtualization. > E.g. if AVX10.2 has a feature that shouldn't be enumerated to guests for > whatever reason, then KVM can't enumerate any "later" features either, > because the only way to hide the problematic AVX10.2 feature is to set the > version to AVX10.1 or lower[2]. But most AVX features are just passed > through and don’t have virtualization controls, so AVX10 should not be > problematic in practice. > > [1] https://cdrdv2.intel.com/v1/dl/getContent/784267 > [2] https://lore.kernel.org/all/Zkz5Ak0PQlAN8DxK@google.com/ > > Signed-off-by: Tao Su <tao1.su@linux.intel.com> > --- > Changelog: > v1 -> v2: > - Directly advertise version 1 because version can never be 0. > - Add and advertise feature bits for the supported vector sizes. > > v1: https://lore.kernel.org/all/20240520022002.1494056-1-tao1.su@linux.intel.com/ > --- > arch/x86/include/asm/cpuid.h | 1 + > arch/x86/kvm/cpuid.c | 21 +++++++++++++++++++-- > arch/x86/kvm/reverse_cpuid.h | 8 ++++++++ > 3 files changed, 28 insertions(+), 2 deletions(-) > > diff --git a/arch/x86/include/asm/cpuid.h b/arch/x86/include/asm/cpuid.h > index 6b122a31da06..aa21c105eef1 100644 > --- a/arch/x86/include/asm/cpuid.h > +++ b/arch/x86/include/asm/cpuid.h > @@ -179,6 +179,7 @@ static __always_inline bool cpuid_function_is_indexed(u32 function) > case 0x1d: > case 0x1e: > case 0x1f: > + case 0x24: > case 0x8000001d: > return true; > } > diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c > index f2f2be5d1141..6717a5b7d9cd 100644 > --- a/arch/x86/kvm/cpuid.c > +++ b/arch/x86/kvm/cpuid.c > @@ -693,7 +693,7 @@ void kvm_set_cpu_caps(void) > > kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX, > F(AVX_VNNI_INT8) | F(AVX_NE_CONVERT) | F(PREFETCHITI) | > - F(AMX_COMPLEX) > + F(AMX_COMPLEX) | F(AVX10) > ); > > kvm_cpu_cap_init_kvm_defined(CPUID_7_2_EDX, > @@ -709,6 +709,10 @@ void kvm_set_cpu_caps(void) > SF(SGX1) | SF(SGX2) | SF(SGX_EDECCSSA) > ); > > + kvm_cpu_cap_init_kvm_defined(CPUID_24_0_EBX, > + F(AVX10_128) | F(AVX10_256) | F(AVX10_512) > + ); > + > kvm_cpu_cap_mask(CPUID_8000_0001_ECX, > F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ | > F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) | > @@ -937,7 +941,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) > switch (function) { > case 0: > /* Limited to the highest leaf implemented in KVM. */ > - entry->eax = min(entry->eax, 0x1fU); > + entry->eax = min(entry->eax, 0x24U); > break; > case 1: > cpuid_entry_override(entry, CPUID_1_EDX); > @@ -1162,6 +1166,19 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) > break; > } > break; > + case 0x24: { No need for the curly braces on the case. But, my suggestion below will change that ;-) > + if (!kvm_cpu_cap_has(X86_FEATURE_AVX10)) { > + entry->eax = entry->ebx = entry->ecx = entry->edx = 0; > + break; > + } > + entry->eax = 0; > + cpuid_entry_override(entry, CPUID_24_0_EBX); > + /* EBX[7:0] hold the AVX10 version; KVM supports version '1'. */ > + entry->ebx |= 1; Ah, rather than hardcode this to '1', I think we should do: u8 avx10_version; if (!kvm_cpu_cap_has(X86_FEATURE_AVX10)) { entry->eax = entry->ebx = entry->ecx = entry->edx = 0; break; } /* * The AVX10 version is encoded in EBX[7:0]. Note, the version * is guaranteed to be >=1 if AVX10 is supported. Note #2, the * version needs to be captured before overriding EBX features! */ avx10_version = min_t(u8, entry->ebx & 0xff, 1); cpuid_entry_override(entry, CPUID_24_0_EBX); entry->ebx |= avx10_version; I.e. use the same approach as limiting the max leaf, which does: entry->eax = min(entry->eax, 0x1fU); Unless I'm misunderstanding how all of this is expected to play out, we're going to need the min_t() code for AVX10.2 anyways, might as well implement it now.
On Fri, Aug 16, 2024 at 07:51:48AM -0700, Sean Christopherson wrote: > On Mon, Jun 03, 2024, Tao Su wrote: > > Advertise AVX10.1 related CPUIDs, i.e. report AVX10 support bit via > > CPUID.(EAX=07H, ECX=01H):EDX[bit 19] and new CPUID leaf 0x24H so that > > guest OS and applications can query the AVX10.1 CPUIDs directly. Intel > > AVX10 represents the first major new vector ISA since the introduction of > > Intel AVX512, which will establish a common, converged vector instruction > > set across all Intel architectures[1]. > > > > AVX10.1 is an early version of AVX10, that enumerates the Intel AVX512 > > instruction set at 128, 256, and 512 bits which is enabled on > > Granite Rapids. I.e., AVX10.1 is only a new CPUID enumeration with no > > VMX capability, Embedded rounding and Suppress All Exceptions (SAE), > > which will be introduced in AVX10.2. > > > > Advertising AVX10.1 is safe because kernel doesn't enable AVX10.1 which is > > I thought there is nothing to enable for AVX10.1? I.e. it's purely a new way to > enumerate support, thus there will never be anything for the kernel to enable. > Yes, AVX10.1 is just a new enumeration way. > > on KVM-only leaf now, just the CPUID checking is changed when using AVX512 > > related instructions, e.g. if using one AVX512 instruction needs to check > > (AVX512 AND AVX512DQ), it can check ((AVX512 AND AVX512DQ) OR AVX10.1) > > after checking XCR0[7:5]. > > > > The versions of AVX10 are expected to be inclusive, e.g. version N+1 is > > a superset of version N. Per the spec, the version can never be 0, just > > advertise AVX10.1 if it's supported in hardware. > > I think it's also worth calling out that advertising AVX10_{128,256,512} needs > to land in the same patch (this patch) as AVX10 (and thus AVX10.1), because > otherwise KVM would advertise an impossible CPU model, e.g. with AVX512 but not > AVX10.1/512, which per "Feature Differences Between Intel® AVX-512 and Intel® AVX10" > should be impossible. > Indeed, that's the reason why I only used one patch, will do in v3, thanks! > > As more and more AVX related CPUIDs are added (it would have resulted in > > around 40-50 CPUID flags when developing AVX10), the versioning approach > > is introduced. But incrementing version numbers are bad for virtualization. > > E.g. if AVX10.2 has a feature that shouldn't be enumerated to guests for > > whatever reason, then KVM can't enumerate any "later" features either, > > because the only way to hide the problematic AVX10.2 feature is to set the > > version to AVX10.1 or lower[2]. But most AVX features are just passed > > through and don’t have virtualization controls, so AVX10 should not be > > problematic in practice. > > > > [1] https://cdrdv2.intel.com/v1/dl/getContent/784267 > > [2] https://lore.kernel.org/all/Zkz5Ak0PQlAN8DxK@google.com/ > > > > Signed-off-by: Tao Su <tao1.su@linux.intel.com> > > --- > > Changelog: > > v1 -> v2: > > - Directly advertise version 1 because version can never be 0. > > - Add and advertise feature bits for the supported vector sizes. > > > > v1: https://lore.kernel.org/all/20240520022002.1494056-1-tao1.su@linux.intel.com/ > > --- > > arch/x86/include/asm/cpuid.h | 1 + > > arch/x86/kvm/cpuid.c | 21 +++++++++++++++++++-- > > arch/x86/kvm/reverse_cpuid.h | 8 ++++++++ > > 3 files changed, 28 insertions(+), 2 deletions(-) > > > > diff --git a/arch/x86/include/asm/cpuid.h b/arch/x86/include/asm/cpuid.h > > index 6b122a31da06..aa21c105eef1 100644 > > --- a/arch/x86/include/asm/cpuid.h > > +++ b/arch/x86/include/asm/cpuid.h > > @@ -179,6 +179,7 @@ static __always_inline bool cpuid_function_is_indexed(u32 function) > > case 0x1d: > > case 0x1e: > > case 0x1f: > > + case 0x24: > > case 0x8000001d: > > return true; > > } > > diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c > > index f2f2be5d1141..6717a5b7d9cd 100644 > > --- a/arch/x86/kvm/cpuid.c > > +++ b/arch/x86/kvm/cpuid.c > > @@ -693,7 +693,7 @@ void kvm_set_cpu_caps(void) > > > > kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX, > > F(AVX_VNNI_INT8) | F(AVX_NE_CONVERT) | F(PREFETCHITI) | > > - F(AMX_COMPLEX) > > + F(AMX_COMPLEX) | F(AVX10) > > ); > > > > kvm_cpu_cap_init_kvm_defined(CPUID_7_2_EDX, > > @@ -709,6 +709,10 @@ void kvm_set_cpu_caps(void) > > SF(SGX1) | SF(SGX2) | SF(SGX_EDECCSSA) > > ); > > > > + kvm_cpu_cap_init_kvm_defined(CPUID_24_0_EBX, > > + F(AVX10_128) | F(AVX10_256) | F(AVX10_512) > > + ); > > + > > kvm_cpu_cap_mask(CPUID_8000_0001_ECX, > > F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ | > > F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) | > > @@ -937,7 +941,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) > > switch (function) { > > case 0: > > /* Limited to the highest leaf implemented in KVM. */ > > - entry->eax = min(entry->eax, 0x1fU); > > + entry->eax = min(entry->eax, 0x24U); > > break; > > case 1: > > cpuid_entry_override(entry, CPUID_1_EDX); > > @@ -1162,6 +1166,19 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) > > break; > > } > > break; > > + case 0x24: { > > No need for the curly braces on the case. But, my suggestion below will change > that ;-) > I got it :-) > > + if (!kvm_cpu_cap_has(X86_FEATURE_AVX10)) { > > + entry->eax = entry->ebx = entry->ecx = entry->edx = 0; > > + break; > > + } > > + entry->eax = 0; > > + cpuid_entry_override(entry, CPUID_24_0_EBX); > > + /* EBX[7:0] hold the AVX10 version; KVM supports version '1'. */ > > + entry->ebx |= 1; > > Ah, rather than hardcode this to '1', I think we should do: > > u8 avx10_version; > > if (!kvm_cpu_cap_has(X86_FEATURE_AVX10)) { > entry->eax = entry->ebx = entry->ecx = entry->edx = 0; > break; > } > > /* > * The AVX10 version is encoded in EBX[7:0]. Note, the version > * is guaranteed to be >=1 if AVX10 is supported. Note #2, the > * version needs to be captured before overriding EBX features! > */ > avx10_version = min_t(u8, entry->ebx & 0xff, 1); > > cpuid_entry_override(entry, CPUID_24_0_EBX); > entry->ebx |= avx10_version; > > I.e. use the same approach as limiting the max leaf, which does: > > entry->eax = min(entry->eax, 0x1fU); > > Unless I'm misunderstanding how all of this is expected to play out, we're going > to need the min_t() code for AVX10.2 anyways, might as well implement it now. Yes, that's better for the upcoming AVX10.2, thanks for the suggestions!
diff --git a/arch/x86/include/asm/cpuid.h b/arch/x86/include/asm/cpuid.h index 6b122a31da06..aa21c105eef1 100644 --- a/arch/x86/include/asm/cpuid.h +++ b/arch/x86/include/asm/cpuid.h @@ -179,6 +179,7 @@ static __always_inline bool cpuid_function_is_indexed(u32 function) case 0x1d: case 0x1e: case 0x1f: + case 0x24: case 0x8000001d: return true; } diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index f2f2be5d1141..6717a5b7d9cd 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -693,7 +693,7 @@ void kvm_set_cpu_caps(void) kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX, F(AVX_VNNI_INT8) | F(AVX_NE_CONVERT) | F(PREFETCHITI) | - F(AMX_COMPLEX) + F(AMX_COMPLEX) | F(AVX10) ); kvm_cpu_cap_init_kvm_defined(CPUID_7_2_EDX, @@ -709,6 +709,10 @@ void kvm_set_cpu_caps(void) SF(SGX1) | SF(SGX2) | SF(SGX_EDECCSSA) ); + kvm_cpu_cap_init_kvm_defined(CPUID_24_0_EBX, + F(AVX10_128) | F(AVX10_256) | F(AVX10_512) + ); + kvm_cpu_cap_mask(CPUID_8000_0001_ECX, F(LAHF_LM) | F(CMP_LEGACY) | 0 /*SVM*/ | 0 /* ExtApicSpace */ | F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) | @@ -937,7 +941,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) switch (function) { case 0: /* Limited to the highest leaf implemented in KVM. */ - entry->eax = min(entry->eax, 0x1fU); + entry->eax = min(entry->eax, 0x24U); break; case 1: cpuid_entry_override(entry, CPUID_1_EDX); @@ -1162,6 +1166,19 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) break; } break; + case 0x24: { + if (!kvm_cpu_cap_has(X86_FEATURE_AVX10)) { + entry->eax = entry->ebx = entry->ecx = entry->edx = 0; + break; + } + entry->eax = 0; + cpuid_entry_override(entry, CPUID_24_0_EBX); + /* EBX[7:0] hold the AVX10 version; KVM supports version '1'. */ + entry->ebx |= 1; + entry->ecx = 0; + entry->edx = 0; + break; + } case KVM_CPUID_SIGNATURE: { const u32 *sigptr = (const u32 *)KVM_SIGNATURE; entry->eax = KVM_CPUID_FEATURES; diff --git a/arch/x86/kvm/reverse_cpuid.h b/arch/x86/kvm/reverse_cpuid.h index 2f4e155080ba..0d17d6b70639 100644 --- a/arch/x86/kvm/reverse_cpuid.h +++ b/arch/x86/kvm/reverse_cpuid.h @@ -17,6 +17,7 @@ enum kvm_only_cpuid_leafs { CPUID_8000_0007_EDX, CPUID_8000_0022_EAX, CPUID_7_2_EDX, + CPUID_24_0_EBX, NR_KVM_CPU_CAPS, NKVMCAPINTS = NR_KVM_CPU_CAPS - NCAPINTS, @@ -46,6 +47,7 @@ enum kvm_only_cpuid_leafs { #define X86_FEATURE_AVX_NE_CONVERT KVM_X86_FEATURE(CPUID_7_1_EDX, 5) #define X86_FEATURE_AMX_COMPLEX KVM_X86_FEATURE(CPUID_7_1_EDX, 8) #define X86_FEATURE_PREFETCHITI KVM_X86_FEATURE(CPUID_7_1_EDX, 14) +#define X86_FEATURE_AVX10 KVM_X86_FEATURE(CPUID_7_1_EDX, 19) /* Intel-defined sub-features, CPUID level 0x00000007:2 (EDX) */ #define X86_FEATURE_INTEL_PSFD KVM_X86_FEATURE(CPUID_7_2_EDX, 0) @@ -55,6 +57,11 @@ enum kvm_only_cpuid_leafs { #define KVM_X86_FEATURE_BHI_CTRL KVM_X86_FEATURE(CPUID_7_2_EDX, 4) #define X86_FEATURE_MCDT_NO KVM_X86_FEATURE(CPUID_7_2_EDX, 5) +/* Intel-defined sub-features, CPUID level 0x00000024:0 (EBX) */ +#define X86_FEATURE_AVX10_128 KVM_X86_FEATURE(CPUID_24_0_EBX, 16) +#define X86_FEATURE_AVX10_256 KVM_X86_FEATURE(CPUID_24_0_EBX, 17) +#define X86_FEATURE_AVX10_512 KVM_X86_FEATURE(CPUID_24_0_EBX, 18) + /* CPUID level 0x80000007 (EDX). */ #define KVM_X86_FEATURE_CONSTANT_TSC KVM_X86_FEATURE(CPUID_8000_0007_EDX, 8) @@ -90,6 +97,7 @@ static const struct cpuid_reg reverse_cpuid[] = { [CPUID_8000_0021_EAX] = {0x80000021, 0, CPUID_EAX}, [CPUID_8000_0022_EAX] = {0x80000022, 0, CPUID_EAX}, [CPUID_7_2_EDX] = { 7, 2, CPUID_EDX}, + [CPUID_24_0_EBX] = { 0x24, 0, CPUID_EBX}, }; /*
Advertise AVX10.1 related CPUIDs, i.e. report AVX10 support bit via CPUID.(EAX=07H, ECX=01H):EDX[bit 19] and new CPUID leaf 0x24H so that guest OS and applications can query the AVX10.1 CPUIDs directly. Intel AVX10 represents the first major new vector ISA since the introduction of Intel AVX512, which will establish a common, converged vector instruction set across all Intel architectures[1]. AVX10.1 is an early version of AVX10, that enumerates the Intel AVX512 instruction set at 128, 256, and 512 bits which is enabled on Granite Rapids. I.e., AVX10.1 is only a new CPUID enumeration with no VMX capability, Embedded rounding and Suppress All Exceptions (SAE), which will be introduced in AVX10.2. Advertising AVX10.1 is safe because kernel doesn't enable AVX10.1 which is on KVM-only leaf now, just the CPUID checking is changed when using AVX512 related instructions, e.g. if using one AVX512 instruction needs to check (AVX512 AND AVX512DQ), it can check ((AVX512 AND AVX512DQ) OR AVX10.1) after checking XCR0[7:5]. The versions of AVX10 are expected to be inclusive, e.g. version N+1 is a superset of version N. Per the spec, the version can never be 0, just advertise AVX10.1 if it's supported in hardware. As more and more AVX related CPUIDs are added (it would have resulted in around 40-50 CPUID flags when developing AVX10), the versioning approach is introduced. But incrementing version numbers are bad for virtualization. E.g. if AVX10.2 has a feature that shouldn't be enumerated to guests for whatever reason, then KVM can't enumerate any "later" features either, because the only way to hide the problematic AVX10.2 feature is to set the version to AVX10.1 or lower[2]. But most AVX features are just passed through and don’t have virtualization controls, so AVX10 should not be problematic in practice. [1] https://cdrdv2.intel.com/v1/dl/getContent/784267 [2] https://lore.kernel.org/all/Zkz5Ak0PQlAN8DxK@google.com/ Signed-off-by: Tao Su <tao1.su@linux.intel.com> --- Changelog: v1 -> v2: - Directly advertise version 1 because version can never be 0. - Add and advertise feature bits for the supported vector sizes. v1: https://lore.kernel.org/all/20240520022002.1494056-1-tao1.su@linux.intel.com/ --- arch/x86/include/asm/cpuid.h | 1 + arch/x86/kvm/cpuid.c | 21 +++++++++++++++++++-- arch/x86/kvm/reverse_cpuid.h | 8 ++++++++ 3 files changed, 28 insertions(+), 2 deletions(-) base-commit: c3f38fa61af77b49866b006939479069cd451173