How to analyze golang memory?

by

The heap profile shows active memory, memory the runtime believes is in use by the go program (ie: hasn’t been collected by the garbage collector). When the GC does collect memory the profile shrinks, but no memory is returned to the system. Your future allocations will try to use memory from the pool of previously collected objects before asking the system for more.

From the outside, this means that your program’s memory use will either be increasing, or staying level. What the outside system presents as the “Resident Size” of your program is the number of bytes of RAM is assigned to your program whether it’s holding in-use go values or collected ones.

The reason why these two numbers are often quite different are because:

  1. The GC collecting memory has no effect on the outside view of the program
  2. Memory fragmentation
  3. The GC only runs when the memory in use doubles the memory in use after the previous GC (by default, see: http://golang.org/pkg/runtime/#pkg-overview)

If you want an accurate breakdown of how Go sees the memory you can use the runtime.ReadMemStats call: http://golang.org/pkg/runtime/#ReadMemStats

Alternatively, since you are using web-based profiling if you can access the profiling data through your browser at: http://10.10.58.118:8601/debug/pprof/ , clicking the heap link will show you the debugging view of the heap profile, which has a printout of a runtime.MemStats structure at the bottom.

The runtime.MemStats documentation (http://golang.org/pkg/runtime/#MemStats) has the explanation of all the fields, but the interesting ones for this discussion are:

  • HeapAlloc: essentially what the profiler is giving you (active heap memory)
  • Alloc: similar to HeapAlloc, but for all go managed memory
  • Sys: the total amount of memory (address space) requested from the OS

There will still be discrepancies between Sys, and what the OS reports because what Go asks of the system, and what the OS gives it are not always the same. Also CGO / syscall (eg: malloc / mmap) memory is not tracked by go.

More Related Contents:

  1. Approximate cost to access various caches and main memory?
  2. What’s the actual effect of successful unaligned accesses on x86?
  3. Order of the code and performance
  4. Why do these goroutines not scale their performance from more concurrent executions?
  5. Redis 10x more memory usage than data
  6. preallocate list in R
  7. Performance of function slice parameter vs global variable?
  8. What is a “cache-friendly” code?
  9. How many CPU cycles are needed for each assembly instruction?
  10. Google app script timeout ~ 5 minutes?
  11. What is the best approach to encapsulate blocking I/O in future-rs?
  12. Why does breaking the “output dependency” of LZCNT matter?
  13. nth fibonacci number in sublinear time
  14. Timing JavaFX Canvas Application
  15. What’s the fastest way to merge/join data.frames in R?
  16. How to optimize vlookup for high search count ? (alternatives to VLOOKUP)
  17. What is the fastest way to get the value of π?
  18. Fast query runs slow in SSRS
  19. Entity Framework query slow, but same SQL in SqlQuery is fast
  20. Java: A two dimensional array is stored in column-major or row-major order?
  21. Bulk inserts taking longer than expected using Dapper
  22. Why are compilers so stupid?
  23. SVN performance after many revisions
  24. In SQLite, do prepared statements really improve performance?
  25. Why is appending to a list bad?
  26. Why is putImageData so slow?
  27. Convert slice of string to slice of pointer to string
  28. Client-side logic OR Server-side logic?
  29. Trouble converting long list of data.frames (~1 million) to single data.frame using do.call and ldply
  30. Projecting a list of lists efficiently in F#

Leave a Comment