diff mbox series

crypto: siphash - use _unaligned version by default

Message ID 20211126143329.2689618-1-arnd@kernel.org (mailing list archive)
State Not Applicable
Delegated to: Herbert Xu
Headers show
Series crypto: siphash - use _unaligned version by default | expand

Commit Message

Arnd Bergmann Nov. 26, 2021, 2:33 p.m. UTC
From: Arnd Bergmann <arnd@arndb.de>

On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
because the ordinary load/store instructions (ldr, ldrh, ldrb) can
tolerate any misalignment of the memory address. However, load/store
double and load/store multiple instructions (ldrd, ldm) may still only
be used on memory addresses that are 32-bit aligned, and so we have to
use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we
may end up with a severe performance hit due to alignment traps that
require fixups by the kernel. Testing shows that this currently happens
with clang-13 but not gcc-11. In theory, any compiler version can
produce this bug or other problems, as we are dealing with undefined
behavior in C99 even on architectures that support this in hardware,
see also https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100363.

Fortunately, the get_unaligned() accessors do the right thing: when
building for ARMv6 or later, the compiler will emit unaligned accesses
using the ordinary load/store instructions (but avoid the ones that
require 32-bit alignment). When building for older ARM, those accessors
will emit the appropriate sequence of ldrb/mov/orr instructions. And on
architectures that can truly tolerate any kind of misalignment, the
get_unaligned() accessors resolve to the leXX_to_cpup accessors that
operate on aligned addresses.

Since the compiler will in fact emit ldrd or ldm instructions when
building this code for ARM v6 or later, the solution is to use the
unaligned accessors unconditionally on architectures where this is
known to be fast. The _aligned version of the hash function is
however still needed to get the best performance on architectures
that cannot do any unaligned access in hardware.

This new version avoids the undefined behavior and should produce
the fastest hash on all architectures we support.

Link: https://lore.kernel.org/linux-arm-kernel/20181008211554.5355-4-ard.biesheuvel@linaro.org/
Reported-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Fixes: 2c956a60778c ("siphash: add cryptographically secure PRF")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
 include/linux/siphash.h | 14 ++++----------
 lib/siphash.c           | 12 ++++++------
 2 files changed, 10 insertions(+), 16 deletions(-)

Comments

Jason A. Donenfeld Nov. 26, 2021, 3:03 p.m. UTC | #1
Hi Arnd,

It looks like Ard's old patch never got picked up so you're dusting it
off. It looks like you're doing two things here -- moving from an
ifndef to a much nicer IS_ENABLED, and changing the logic a bit. In
trying to understand the logic part, I changed this in my buffer:

-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
-       if (!IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
+       if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+           !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
                return __hsiphash_unaligned(data, len, key);
        return ___hsiphash_aligned(data, len, key);

into this:

-       if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
-           !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
+       if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+           !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
                return __hsiphash_unaligned(data, len, key);
        return ___hsiphash_aligned(data, len, key);

This way I can actually think about what's happening here.

So with the old one, we use the faster aligned version if *either* the
CPU has efficient unaligned access OR the bytes are statically known
to be aligned. This seems sensible.

On the new one, we use the faster aligned version if *both* the bytes
are statically known to be aligned (ok) AND the CPU doesn't actually
support efficient unaligned accesses (?). This seems kind of weird.

It also means that CPUs with fast aligned accesses wind up calling the
slower code path in some cases. Is your supposition that the compiler
will always optimize the slow codepath to the fast one if the CPU it's
compiling for supports that? Have you tested this on all platforms?

Would it make sense to instead just fix clang-13? Or even to just get
rid of CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS for armv6 or undef
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS for armv6 just in this file or
maybe less messy, split CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS into
two ifdefs that more sense for our usage?

Jason
Arnd Bergmann Nov. 26, 2021, 3:17 p.m. UTC | #2
On Fri, Nov 26, 2021 at 4:03 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hi Arnd,
>
> It looks like Ard's old patch never got picked up so you're dusting it
> off. It looks like you're doing two things here -- moving from an
> ifndef to a much nicer IS_ENABLED, and changing the logic a bit. In
> trying to understand the logic part, I changed this in my buffer:

I actually found the issue independently and came up with this patch
before Ard pointed me to his patch, I mainly took the description of the
problem from him, as his explanation was already well written.

> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> -       if (!IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
> +       if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
> +           !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
>                 return __hsiphash_unaligned(data, len, key);
>         return ___hsiphash_aligned(data, len, key);
>
> into this:
>
> -       if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
> -           !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
> +       if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
> +           !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
>                 return __hsiphash_unaligned(data, len, key);
>         return ___hsiphash_aligned(data, len, key);
>
> This way I can actually think about what's happening here.
>
> So with the old one, we use the faster aligned version if *either* the
> CPU has efficient unaligned access OR the bytes are statically known
> to be aligned. This seems sensible.
>
> On the new one, we use the faster aligned version if *both* the bytes
> are statically known to be aligned (ok) AND the CPU doesn't actually
> support efficient unaligned accesses (?). This seems kind of weird.

Yes, this is intentional. The point is that __hsiphash_unaligned() is
the portable version that works with any alignment on any architecture,
while __hsiphash_aligned() is either identical, or may only be called
with aligned data. Passing an unaligned pointer into this function triggers
undefined behavior in C99, which is how it broke on armv7, but in fact
any compiler might optimize this function based on "knowing" that
the lower address bits are zero.

> It also means that CPUs with fast aligned accesses wind up calling the
> slower code path in some cases. Is your supposition that the compiler
> will always optimize the slow codepath to the fast one if the CPU it's
> compiling for supports that? Have you tested this on all platforms?

I have not tested this specific patch on all platforms, but I did
extensive testing of the get_unaligned()/put_unaligned() helpers
in my rewrite earlier this year[1], making sure that these are NOPs
on all the important architectures, and that they prevent the use
of trapping ldrd/ldm instructions on ARMv6/ARMv7.

> Would it make sense to instead just fix clang-13? Or even to just get
> rid of CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS for armv6 or undef
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS for armv6 just in this file or
> maybe less messy, split CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS into
> two ifdefs that more sense for our usage?

Clang is actually doing the right thing here, it may be considered a missed
optimization that gcc uses two loads instead of a combined ldm or ldrd ;-)

FWIW, the bug that we saw in the decompressor relying on data alignment on x86
earlier this year only happened on gcc.

      Arnd

[1] https://lkml.org/lkml/2021/5/7/775
Jason A. Donenfeld Nov. 26, 2021, 3:26 p.m. UTC | #3
On Fri, Nov 26, 2021 at 10:18 AM Arnd Bergmann <arnd@kernel.org> wrote:
> I have not tested this specific patch on all platforms, but I did
> extensive testing of the get_unaligned()/put_unaligned() helpers
> in my rewrite earlier this year[1], making sure that these are NOPs
> on all the important architectures, and that they prevent the use
> of trapping ldrd/ldm instructions on ARMv6/ARMv7.

If you're confident that the codegen doesn't change for ARMv8 and
x86{,_64}, then:

Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>

We should probably Cc stable@, right?

I'm preparing a patchset for DaveM's stable tree today, and I can take
this in that patchset.

Jason
Ard Biesheuvel Nov. 26, 2021, 3:34 p.m. UTC | #4
On Fri, 26 Nov 2021 at 16:02, Arnd Bergmann <arnd@kernel.org> wrote:
>
> From: Arnd Bergmann <arnd@arndb.de>
>
> On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> because the ordinary load/store instructions (ldr, ldrh, ldrb) can
> tolerate any misalignment of the memory address. However, load/store
> double and load/store multiple instructions (ldrd, ldm) may still only
> be used on memory addresses that are 32-bit aligned, and so we have to
> use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we
> may end up with a severe performance hit due to alignment traps that
> require fixups by the kernel. Testing shows that this currently happens
> with clang-13 but not gcc-11. In theory, any compiler version can
> produce this bug or other problems, as we are dealing with undefined
> behavior in C99 even on architectures that support this in hardware,
> see also https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100363.
>
> Fortunately, the get_unaligned() accessors do the right thing: when
> building for ARMv6 or later, the compiler will emit unaligned accesses
> using the ordinary load/store instructions (but avoid the ones that
> require 32-bit alignment). When building for older ARM, those accessors
> will emit the appropriate sequence of ldrb/mov/orr instructions. And on
> architectures that can truly tolerate any kind of misalignment, the
> get_unaligned() accessors resolve to the leXX_to_cpup accessors that
> operate on aligned addresses.
>
> Since the compiler will in fact emit ldrd or ldm instructions when
> building this code for ARM v6 or later, the solution is to use the
> unaligned accessors unconditionally on architectures where this is
> known to be fast. The _aligned version of the hash function is
> however still needed to get the best performance on architectures
> that cannot do any unaligned access in hardware.
>
> This new version avoids the undefined behavior and should produce
> the fastest hash on all architectures we support.
>
> Link: https://lore.kernel.org/linux-arm-kernel/20181008211554.5355-4-ard.biesheuvel@linaro.org/
> Reported-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Fixes: 2c956a60778c ("siphash: add cryptographically secure PRF")
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>

Acked-by: Ard Biesheuvel <ardb@kernel.org>

> ---
>  include/linux/siphash.h | 14 ++++----------
>  lib/siphash.c           | 12 ++++++------
>  2 files changed, 10 insertions(+), 16 deletions(-)
>
> diff --git a/include/linux/siphash.h b/include/linux/siphash.h
> index bf21591a9e5e..0cda61855d90 100644
> --- a/include/linux/siphash.h
> +++ b/include/linux/siphash.h
> @@ -27,9 +27,7 @@ static inline bool siphash_key_is_zero(const siphash_key_t *key)
>  }
>
>  u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key);
> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key);
> -#endif
>
>  u64 siphash_1u64(const u64 a, const siphash_key_t *key);
>  u64 siphash_2u64(const u64 a, const u64 b, const siphash_key_t *key);
> @@ -82,10 +80,9 @@ static inline u64 ___siphash_aligned(const __le64 *data, size_t len,
>  static inline u64 siphash(const void *data, size_t len,
>                           const siphash_key_t *key)
>  {
> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> -       if (!IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT))
> +       if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
> +           !IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT))
>                 return __siphash_unaligned(data, len, key);
> -#endif
>         return ___siphash_aligned(data, len, key);
>  }
>
> @@ -96,10 +93,8 @@ typedef struct {
>
>  u32 __hsiphash_aligned(const void *data, size_t len,
>                        const hsiphash_key_t *key);
> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u32 __hsiphash_unaligned(const void *data, size_t len,
>                          const hsiphash_key_t *key);
> -#endif
>
>  u32 hsiphash_1u32(const u32 a, const hsiphash_key_t *key);
>  u32 hsiphash_2u32(const u32 a, const u32 b, const hsiphash_key_t *key);
> @@ -135,10 +130,9 @@ static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len,
>  static inline u32 hsiphash(const void *data, size_t len,
>                            const hsiphash_key_t *key)
>  {
> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> -       if (!IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
> +       if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
> +           !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
>                 return __hsiphash_unaligned(data, len, key);
> -#endif
>         return ___hsiphash_aligned(data, len, key);
>  }
>
> diff --git a/lib/siphash.c b/lib/siphash.c
> index a90112ee72a1..72b9068ab57b 100644
> --- a/lib/siphash.c
> +++ b/lib/siphash.c
> @@ -49,6 +49,7 @@
>         SIPROUND; \
>         return (v0 ^ v1) ^ (v2 ^ v3);
>
> +#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key)
>  {
>         const u8 *end = data + len - (len % sizeof(u64));
> @@ -80,8 +81,8 @@ u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key)
>         POSTAMBLE
>  }
>  EXPORT_SYMBOL(__siphash_aligned);
> +#endif
>
> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key)
>  {
>         const u8 *end = data + len - (len % sizeof(u64));
> @@ -113,7 +114,6 @@ u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key)
>         POSTAMBLE
>  }
>  EXPORT_SYMBOL(__siphash_unaligned);
> -#endif
>
>  /**
>   * siphash_1u64 - compute 64-bit siphash PRF value of a u64
> @@ -250,6 +250,7 @@ EXPORT_SYMBOL(siphash_3u32);
>         HSIPROUND; \
>         return (v0 ^ v1) ^ (v2 ^ v3);
>
> +#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
>  {
>         const u8 *end = data + len - (len % sizeof(u64));
> @@ -280,8 +281,8 @@ u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
>         HPOSTAMBLE
>  }
>  EXPORT_SYMBOL(__hsiphash_aligned);
> +#endif
>
> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u32 __hsiphash_unaligned(const void *data, size_t len,
>                          const hsiphash_key_t *key)
>  {
> @@ -313,7 +314,6 @@ u32 __hsiphash_unaligned(const void *data, size_t len,
>         HPOSTAMBLE
>  }
>  EXPORT_SYMBOL(__hsiphash_unaligned);
> -#endif
>
>  /**
>   * hsiphash_1u32 - compute 64-bit hsiphash PRF value of a u32
> @@ -418,6 +418,7 @@ EXPORT_SYMBOL(hsiphash_4u32);
>         HSIPROUND; \
>         return v1 ^ v3;
>
> +#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
>  {
>         const u8 *end = data + len - (len % sizeof(u32));
> @@ -438,8 +439,8 @@ u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
>         HPOSTAMBLE
>  }
>  EXPORT_SYMBOL(__hsiphash_aligned);
> +#endif
>
> -#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
>  u32 __hsiphash_unaligned(const void *data, size_t len,
>                          const hsiphash_key_t *key)
>  {
> @@ -461,7 +462,6 @@ u32 __hsiphash_unaligned(const void *data, size_t len,
>         HPOSTAMBLE
>  }
>  EXPORT_SYMBOL(__hsiphash_unaligned);
> -#endif
>
>  /**
>   * hsiphash_1u32 - compute 32-bit hsiphash PRF value of a u32
> --
> 2.29.2
>
Arnd Bergmann Nov. 26, 2021, 8:19 p.m. UTC | #5
On Fri, Nov 26, 2021 at 4:26 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> On Fri, Nov 26, 2021 at 10:18 AM Arnd Bergmann <arnd@kernel.org> wrote:
> > I have not tested this specific patch on all platforms, but I did
> > extensive testing of the get_unaligned()/put_unaligned() helpers
> > in my rewrite earlier this year[1], making sure that these are NOPs
> > on all the important architectures, and that they prevent the use
> > of trapping ldrd/ldm instructions on ARMv6/ARMv7.
>
> If you're confident that the codegen doesn't change for ARMv8 and
> x86{,_64}, then:
>
> Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>

Thanks

> We should probably Cc stable@, right?

Yes, I meant to add that but forgot.

> I'm preparing a patchset for DaveM's stable tree today, and I can take
> this in that patchset.

It's not urgent, but I think we need to fix it and get it into stable kernels
at some point. I'm happy with whatever timing works for you.

         Arnd
diff mbox series

Patch

diff --git a/include/linux/siphash.h b/include/linux/siphash.h
index bf21591a9e5e..0cda61855d90 100644
--- a/include/linux/siphash.h
+++ b/include/linux/siphash.h
@@ -27,9 +27,7 @@  static inline bool siphash_key_is_zero(const siphash_key_t *key)
 }
 
 u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key);
-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key);
-#endif
 
 u64 siphash_1u64(const u64 a, const siphash_key_t *key);
 u64 siphash_2u64(const u64 a, const u64 b, const siphash_key_t *key);
@@ -82,10 +80,9 @@  static inline u64 ___siphash_aligned(const __le64 *data, size_t len,
 static inline u64 siphash(const void *data, size_t len,
 			  const siphash_key_t *key)
 {
-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
-	if (!IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT))
+	if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+	    !IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT))
 		return __siphash_unaligned(data, len, key);
-#endif
 	return ___siphash_aligned(data, len, key);
 }
 
@@ -96,10 +93,8 @@  typedef struct {
 
 u32 __hsiphash_aligned(const void *data, size_t len,
 		       const hsiphash_key_t *key);
-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u32 __hsiphash_unaligned(const void *data, size_t len,
 			 const hsiphash_key_t *key);
-#endif
 
 u32 hsiphash_1u32(const u32 a, const hsiphash_key_t *key);
 u32 hsiphash_2u32(const u32 a, const u32 b, const hsiphash_key_t *key);
@@ -135,10 +130,9 @@  static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len,
 static inline u32 hsiphash(const void *data, size_t len,
 			   const hsiphash_key_t *key)
 {
-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
-	if (!IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
+	if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
+	    !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
 		return __hsiphash_unaligned(data, len, key);
-#endif
 	return ___hsiphash_aligned(data, len, key);
 }
 
diff --git a/lib/siphash.c b/lib/siphash.c
index a90112ee72a1..72b9068ab57b 100644
--- a/lib/siphash.c
+++ b/lib/siphash.c
@@ -49,6 +49,7 @@ 
 	SIPROUND; \
 	return (v0 ^ v1) ^ (v2 ^ v3);
 
+#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key)
 {
 	const u8 *end = data + len - (len % sizeof(u64));
@@ -80,8 +81,8 @@  u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key)
 	POSTAMBLE
 }
 EXPORT_SYMBOL(__siphash_aligned);
+#endif
 
-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key)
 {
 	const u8 *end = data + len - (len % sizeof(u64));
@@ -113,7 +114,6 @@  u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key)
 	POSTAMBLE
 }
 EXPORT_SYMBOL(__siphash_unaligned);
-#endif
 
 /**
  * siphash_1u64 - compute 64-bit siphash PRF value of a u64
@@ -250,6 +250,7 @@  EXPORT_SYMBOL(siphash_3u32);
 	HSIPROUND; \
 	return (v0 ^ v1) ^ (v2 ^ v3);
 
+#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
 {
 	const u8 *end = data + len - (len % sizeof(u64));
@@ -280,8 +281,8 @@  u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
 	HPOSTAMBLE
 }
 EXPORT_SYMBOL(__hsiphash_aligned);
+#endif
 
-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u32 __hsiphash_unaligned(const void *data, size_t len,
 			 const hsiphash_key_t *key)
 {
@@ -313,7 +314,6 @@  u32 __hsiphash_unaligned(const void *data, size_t len,
 	HPOSTAMBLE
 }
 EXPORT_SYMBOL(__hsiphash_unaligned);
-#endif
 
 /**
  * hsiphash_1u32 - compute 64-bit hsiphash PRF value of a u32
@@ -418,6 +418,7 @@  EXPORT_SYMBOL(hsiphash_4u32);
 	HSIPROUND; \
 	return v1 ^ v3;
 
+#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
 {
 	const u8 *end = data + len - (len % sizeof(u32));
@@ -438,8 +439,8 @@  u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
 	HPOSTAMBLE
 }
 EXPORT_SYMBOL(__hsiphash_aligned);
+#endif
 
-#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
 u32 __hsiphash_unaligned(const void *data, size_t len,
 			 const hsiphash_key_t *key)
 {
@@ -461,7 +462,6 @@  u32 __hsiphash_unaligned(const void *data, size_t len,
 	HPOSTAMBLE
 }
 EXPORT_SYMBOL(__hsiphash_unaligned);
-#endif
 
 /**
  * hsiphash_1u32 - compute 32-bit hsiphash PRF value of a u32