latency vs throughput in intel intrinsics

by

For a much more complete picture of CPU performance, see Agner Fog’s microarchitecture guide and instruction tables. (Also his Optimizing C++ and Optimizing Assembly guides are excellent). See also other links in the x86 tag wiki, especially Intel’s optimization manual.

See also

Latency and throughput for a single instruction are not actually enough to get a useful picture for a loop that uses a mix of vector instructions. Those numbers don’t tell you which intrinsics (asm instructions) compete with each other for throughput resources (i.e. whether they need the same execution port or not). They’re only sufficient for super-simple loops that e.g. load / do one thing / store, or e.g. sum an array with _mm_add_ps or _mm_add_epi32.

You can use multiple accumulators to get more instruction-level parallelism, but you’re still only using one intrinsic so you do have enough information to see that e.g. CPUs before Skylake can only sustain a throughput of one _mm_add_ps per clock, while SKL can start two per clock cycle (reciprocal throughput of one per 0.5c). It can run ADDPS on both its fully-pipelined FMA execution units, instead of having a single dedicated FP-add unit, hence the better throughput but worse latency than Haswell (3c lat, one per 1c tput).

Since _mm_add_ps has a latency of 4 cycles on Skylake, that means 8 vector-FP add operations can be in flight at once. So you need 8 independent vector accumulators (which you add to each other at the end) to expose that much parallelism. (e.g. manually unroll your loop with 8 separate __m256 sum0, sum1, ... variables. Compiler-driven unrolling (compile with -funroll-loops -ffast-math) will often use the same register, but loop overhead wasn’t the problem).

Those numbers also leave out the third major dimension of Intel CPU performance: fused-domain uop throughput. Most instructions decode to a single uop, but some decode to multiple uops. (Especially the SSE4.2 string instructions like the _mm_cmpestrc you mentioned: PCMPESTRI is 8 uops on Skylake). Even if there’s no bottleneck on any specific execution port, you can still bottleneck on the frontend’s ability to keep the out-of-order core fed with work to do. Intel Sandybridge-family CPUs can issue up to 4 fused-domain uops per clock, and in practice can often come close to that when other bottlenecks don’t occur. (See Is performance reduced when executing loops whose uop count is not a multiple of processor width? for some interesting best-case frontend throughput tests for different loop sizes.) Since load/store instructions use different execution ports than ALU instructions, this can be the bottleneck when data is hot in L1 cache.

And unless you look at the compiler-generated asm, you won’t know how many extra MOVDQA instructions the compiler had to use to copy data between registers, to work around the fact that without AVX, most instructions replace their first source register with the result. (i.e. destructive destination). You also won’t know about loop overhead from any scalar operations in the loop.

I think I have a decent understanding of the difference between latency and throughput

Your guesses don’t seem to make sense, so you’re definitely missing something.

CPUs are pipelined, and so are the execution units inside them. A “fully pipelined” execution unit can start a new operation every cycle (throughput = one per clock)

  • (reciprocal) Throughput is how often an operation can start when no data dependencies force it to wait, e.g. one per 7 cycles for this instruction.

  • Latency is how long it takes for the results of one operation to be ready, and usually matters only when it’s part of a loop-carried dependency chain.

    If the next iteration of a loop operates independently from the previous, then out-of-order execution can “see” far enough ahead to find the instruction-level parallelism between two iterations and keep itself busy, bottlenecking only on throughput.

More Related Contents:

  1. What is the best way to set a register to zero in x86 assembly: xor, mov or and?
  2. INC instruction vs ADD 1: Does it matter?
  3. Is performance reduced when executing loops whose uop count is not a multiple of processor width?
  4. Why is this SSE code 6 times slower without VZEROUPPER on Skylake?
  5. Why does breaking the “output dependency” of LZCNT matter?
  6. Is there a penalty when base+offset is in a different page than the base?
  7. Branch alignment for loops involving micro-coded instructions on Intel SnB-family CPUs
  8. What methods can be used to efficiently extend instruction length on modern x86?
  9. Non-temporal loads and the hardware prefetcher, do they work together?
  10. Why is SSE scalar sqrt(x) slower than rsqrt(x) * x?
  11. Which Intel microarchitecture introduced the ADC reg,0 single-uop special case?
  12. Can modern x86 implementations store-forward from more than one prior store?
  13. Assembly – How to score a CPU instruction by latency and throughput
  14. Is it useful to use VZEROUPPER if your program+libraries contain no SSE instructions?
  15. Modern x86 cost model
  16. Cycles/cost for L1 Cache hit vs. Register on x86?
  17. Efficient sse shuffle mask generation for left-packing byte elements
  18. How can the rep stosb instruction execute faster than the equivalent loop?
  19. How many CPU cycles are needed for each assembly instruction?
  20. Understanding the impact of lfence on a loop with two long dependency chains, for increasing lengths
  21. What is the purpose of the EBP frame pointer register?
  22. Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures?
  23. clflush to invalidate cache line via C function
  24. Size of store buffers on Intel hardware? What exactly is a store buffer?
  25. How to merge a scalar into a vector without the compiler wasting an instruction zeroing upper elements? Design limitation in Intel’s intrinsics?
  26. Header files for x86 SIMD intrinsics
  27. How are cache memories shared in multicore Intel CPUs?
  28. Return address prediction stack buffer vs stack-stored return address?
  29. When should we use prefetch?
  30. Relative performance of x86 inc vs. add instruction

Leave a Comment