How to sum __m256 horizontally?

by

This version should be optimal for both Intel Sandy/Ivy Bridge and AMD Bulldozer, and later CPUs.

// x = ( x7, x6, x5, x4, x3, x2, x1, x0 )
float sum8(__m256 x) {
    // hiQuad = ( x7, x6, x5, x4 )
    const __m128 hiQuad = _mm256_extractf128_ps(x, 1);
    // loQuad = ( x3, x2, x1, x0 )
    const __m128 loQuad = _mm256_castps256_ps128(x);
    // sumQuad = ( x3 + x7, x2 + x6, x1 + x5, x0 + x4 )
    const __m128 sumQuad = _mm_add_ps(loQuad, hiQuad);
    // loDual = ( -, -, x1 + x5, x0 + x4 )
    const __m128 loDual = sumQuad;
    // hiDual = ( -, -, x3 + x7, x2 + x6 )
    const __m128 hiDual = _mm_movehl_ps(sumQuad, sumQuad);
    // sumDual = ( -, -, x1 + x3 + x5 + x7, x0 + x2 + x4 + x6 )
    const __m128 sumDual = _mm_add_ps(loDual, hiDual);
    // lo = ( -, -, -, x0 + x2 + x4 + x6 )
    const __m128 lo = sumDual;
    // hi = ( -, -, -, x1 + x3 + x5 + x7 )
    const __m128 hi = _mm_shuffle_ps(sumDual, sumDual, 0x1);
    // sum = ( -, -, -, x0 + x1 + x2 + x3 + x4 + x5 + x6 + x7 )
    const __m128 sum = _mm_add_ss(lo, hi);
    return _mm_cvtss_f32(sum);
}

haddps is not efficient on any CPU; the best you can do is one shuffle (to extract the high half) and one add, repeat until one element left. Narrowing to 128-bit as the first step benefits AMD before Zen2, and is not a bad thing anywhere.

See Fastest way to do horizontal SSE vector sum on x86 for more details about efficiency.

More Related Contents:

  1. practical BigNum AVX/SSE possible?
  2. Where is Clang’s ‘_mm256_pow_ps’ intrinsic?
  3. The Effect of Architecture When Using SSE / AVX Intrinisics
  4. SIMD math libraries for SSE and AVX
  5. AVX2 what is the most efficient way to pack left based on a mask?
  6. Why is this SSE code 6 times slower without VZEROUPPER on Skylake?
  7. Why doesn’t gcc resolve _mm256_loadu_pd as single vmovupd?
  8. How to solve the 32-byte-alignment issue for AVX load/store operations?
  9. What are the best instruction sequences to generate vector constants on the fly?
  10. is there an inverse instruction to the movemask instruction in intel avx2?
  11. Do any JVM’s JIT compilers generate code that uses vectorized floating point instructions?
  12. Loading 8 chars from memory into an __m256 variable as packed single precision floats
  13. Using AVX CPU instructions: Poor performance without “/arch:AVX”
  14. Fast vectorized rsqrt and reciprocal with SSE/AVX depending on precision
  15. Fastest way to compute absolute value using SSE
  16. Loop unrolling to achieve maximum throughput with Ivy Bridge and Haswell
  17. Header files for x86 SIMD intrinsics
  18. How to detect SSE/SSE2/AVX/AVX2/AVX-512/AVX-128-FMA/KCVI availability at compile-time?
  19. AVX/SSE version of xorshift128+
  20. Per-element atomicity of vector load/store and gather/scatter?
  21. Fastest method to calculate sum of all packed 32-bit integers using AVX512 or AVX2
  22. latency vs throughput in intel intrinsics
  23. How to use Fused Multiply-Add (FMA) instructions with SSE/AVX
  24. C++ error: ‘_mm_sin_ps’ was not declared in this scope
  25. Get sum of values stored in __m256d with SSE/AVX
  26. Get member of __m128 by index?
  27. Can PTEST be used to test if two registers are both zero or some other condition?
  28. Visual Studio 2017: _mm_load_ps often compiled to movups
  29. Fastest way to set __m256 value to all ONE bits
  30. Preventing GCC from automatically using AVX and FMA instructions when compiled with -mavx and -mfma

Leave a Comment