Languages and runtimesDive 03 of 06
Go’s garbage collector, from GOGC to the cache line
Follow a 64-byte cache entry through allocation, concurrent marking and sweeping. Break a collector, tune a heap goal, read real runtime output, and see why Green Tea changes the order in which memory is touched.
- Levels
- 8
- Reading
- 45 min
- Depth
- Idea to Silicon
- Published
Who gets to say an object is dead?
Your service keeps a linked cache of recently used results. Each request may add an entry, replace an old one, or hand an existing entry to another goroutine. When a request returns, some entries must survive because the cache still needs them. Others have no remaining reader. The hard question is who knows when every possible reader has finished.
Here is the entry we will follow. On amd64, its pointer takes eight bytes, its key takes eight, and its payload takes forty-eight. The struct is 64 bytes, with no extra padding. Inserting it publishes a pointer through the global head; removing that path may eventually make it garbage. Synchronization around a real shared cache is essential but omitted here so we can examine lifetime separately.
type Entry struct {
next *Entry // 8 bytes on amd64
key uint64 // 8 bytes
payload [48]byte // 48 bytes; no pointers
}
var head *Entry
func insert(key uint64) {
e := &Entry{key: key, next: head}
head = e
}The entry’s lifetime is a graph property
- Allocate 64 zeroed bytes in a scan span
- Publish cache → entry → next
- Unlink Last strong path disappears
- Reclaim later After a mark cycle proves it dead
A tracing garbage collector replaces “the programmer says this allocation is dead” with “there is no strong pointer path from anything the program can still use.” It starts at A reference the collector starts from rather than discovers by following another heap object's fields: for example, a live stack pointer or a pointer in a global variable., follows references, and preserves everything reachable. A self-contained cycle is no obstacle: if neither object is reachable from a root, both can be reclaimed. Conversely, a forgotten entry linked from a global cache is live to the collector forever, even if nobody intends to read it again.
This solves a coordination problem. The request handler, cache and background task do not need to agree on an explicit last free. They still need to agree on who may mutate an object and when. Garbage collection gives memory lifetime safety for ordinary Go references; it does not make a racing update safe, bound a cache, or close a socket at the right time.
Which bill do you want to pay?
Memory management always has a bill. Manual allocation makes the program describe destruction explicitly. Ownership systems move much of that reasoning into the type system and scope rules. Reference counting spreads bookkeeping over reference updates. Tracing collectors periodically inspect the reachable graph and need room for allocations made between inspections. The right comparison includes allocator cost, locality and programmer constraints, not just pause duration.
| Strategy | Reclamation decision | What it buys | What it costs |
|---|---|---|---|
| Manual / ownership | Explicit free or owner destruction | Control over lifetime; scope-based resource cleanup | Ownership constraints; unsafe manual code risks dangling pointers and leaks. Allocator work remains. |
| Reference counting | Strong reference count reaches zero | Usually prompt reclamation without tracing the whole live graph | Reference-update traffic, possible atomic operations and destruction cascades; cycles need another mechanism. |
| Moving / generational tracing | Trace selected regions or generations, often relocate survivors | Fast young allocation, compaction and locality; exploit short lifetimes | Remember cross-generation edges and update or mediate references; evacuation needs space and coordination. |
| Go’s tracing mark-sweep | Mark reachable objects; sweep unmarked slots | Concurrent tracing without relocating heap objects; simple age-independent model | Heap headroom, marking CPU, pointer-write barriers and fragmentation without heap compaction. |
For concrete contrasts, HotSpot G1 collects region sets and evacuates objects; ZGC uses relocation and reference barriers to keep pauses short. Modern ZGC is itself generational. .NET organizes managed objects into generations and can compact them, with different treatment for large objects and optional background collection. Each design balances latency, throughput and memory under different constraints. See the G1 description, generational ZGC design and .NET GC fundamentals.
Go’s production collector, including Green Tea, is concurrent, non-generational and non-moving for heap objects. It avoids a heap evacuation phase and spends substantial engineering effort keeping global pauses short. “Concurrent” means application goroutines run during much of its work; “parallel” means several processors can do collector work together. The mutator, the application code that changes the object graph, shares CPU and memory bandwidth with the collector.
There is one useful boundary on “non-moving.” Go’s goroutine stacks can grow by copying, so pointers into stack storage may be updated. That is separate from moving live heap allocations to compact the heap. The stack-growth level in the Go concurrency dive follows that machinery. Here we follow our entry once it has escaped to the heap. Its address can stay fixed while the runtime decides whether its slot is still occupied.
The collector proves reachability, not usefulness. A small live set and a well-bounded ownership graph are performance features even in a garbage-collected language.
The knobs and the evidence
Our cache service may first show higher CPU, reduced request throughput, or a tail-latency spike during an allocation burst, even while global pauses remain short. Before changing a knob, separate three quantities: bytes allocated per second, bytes still reachable after collection, and bytes whose pointer fields need scanning. They answer different questions. A service can allocate furiously yet retain little, or allocate slowly while scanning a large pointer-rich cache every cycle.
GOGC controls the percentage-based heap-growth policy. Its default is 100. Raising it normally gives the heap more headroom and reduces collection frequency; lowering it normally saves heap space at the cost of more frequent work. It does not specify a pause duration or a collection interval. GOGC=off disables this percentage trigger. A finite memory limit can still cause collections.
GOMEMLIMIT, introduced in Go 1.19, supplies a soft limit for memory managed by the runtime. It accepts values such as 768MiB. It is not an RSS limit, a cgroup limit, or a promise that allocation fails at exactly that number. The runtime can spend more collector effort and return free pages more aggressively as it approaches the budget. We will derive the headroom calculation after opening the collector.
GOGC=100 GOMEMLIMIT=768MiB ./server
GODEBUG=gctrace=1 ./server 2>gc.log
go build -gcflags=-m ./...
go build -gcflags='all=-m=2' ./... # includes dependenciesimport "runtime/debug"
previousPercent := debug.SetGCPercent(150)
previousLimit := debug.SetMemoryLimit(768 << 20)
_, _ = previousPercent, previousLimit
// Negative SetGCPercent disables percentage-triggered GC.
// A finite memory limit still applies.
// SetMemoryLimit(-1) reads the setting without changing it.The corresponding APIs, runtime/debug.SetGCPercent and SetMemoryLimit, return the previous setting. This lets an operator or controller change policy without recompilation. These settings affect the whole process, so give one controller responsibility for changing them. A negative memory-limit argument queries the current value; it does not disable the limit. The API documentation defines the accounting and special values precisely: SetGCPercent and SetMemoryLimit.
Read the line before blaming the pause
GODEBUG=gctrace=1 prints one summary per collection. The example below is a real Go 1.27.3 linux/amd64 line, captured from a program retaining 4 MiB byte arrays and calling runtime.GC(). It is intentionally a forced collection so its trigger is unambiguous. Its timings describe that short run, not typical production latency. Try changing a field or pasting another line in the same format.
Parser for the format shown, including an optional “(forced)” suffix. Runtime diagnostics are version-dependent, not a stable interchange format. “MB” is the runtime’s label for values shifted by 20 bits, effectively MiB; small values truncate to zero.
The three wall-clock durations are sweep termination, concurrent marking, and mark termination. Only the first and last are global stop-the-world portions. The five CPU durations split concurrent marking into assists, background workers and idle workers, between the two termination costs. With multiple processors, aggregate CPU time can exceed elapsed wall time. The percentage near the beginning reports the share of available CPU since process start used by GC, excluding idle marking. Go computes it as (GCTotalTime − GCIdleTime) / TotalTime in runtime/mgc.go.
The heap triplet means heap at collection start, heap at collection end, and live heap. Allocation can continue during marking, so the middle number can exceed the first. The final number need not represent every byte the process retains: stacks, metadata and other runtime memory have separate accounting. Likewise, a printed zero for globals or stacks may just mean less than one MiB after truncation. The runtime prints “MB” but shifts byte counts by twenty bits. Consult the versioned format definition before writing a parser.
For a monitoring system, prefer runtime/metrics to parsing stderr. Read live heap, goal and scan bytes separately, then take differences of cumulative CPU counters across an interval. A high mark-assist rate tells you allocation is making goroutines pay collector debt. Histograms such as /sched/pauses/total/gc:seconds describe pauses; they are not interchangeable with total GC CPU. Check metric kinds before reading values, and use metrics.All() to discover the version’s schema.
package main
import (
"fmt"
"runtime/metrics"
)
func main() {
samples := []metrics.Sample{
{Name: "/gc/heap/live:bytes"},
{Name: "/gc/heap/goal:bytes"},
{Name: "/gc/scan/heap:bytes"},
{Name: "/gc/scan/stack:bytes"},
{Name: "/gc/scan/globals:bytes"},
{Name: "/cpu/classes/gc/mark/assist:cpu-seconds"},
{Name: "/gc/limiter/last-enabled:gc-cycle"},
}
metrics.Read(samples)
for _, s := range samples {
switch s.Value.Kind() {
case metrics.KindUint64:
fmt.Println(s.Name, s.Value.Uint64())
case metrics.KindFloat64:
fmt.Println(s.Name, s.Value.Float64())
}
}
}Control lifetime before tuning collection
The cheapest garbage is an allocation you avoid. go build -gcflags=-m reports compiler escape decisions for the packages being built. Add all= to include dependencies, or -m=2 for more detail. Escape is a lifetime proof: can this storage safely remain in a frame, or must it survive elsewhere? Taking an address alone is not proof of escape. Returning a pointer is also context-dependent because inlining can expose a short lifetime to the caller.
The cheapest garbage is none · Go 1.27.3, inlining disabled
func Local() int {
x := 42
return *(&x)
}No “moved to heap” diagnostic for x.
The optimized Local function returns the constant 42.Taking an address does not force a heap allocation. The pointer stays within the call; optimization removes the local entirely.
func Escape() *int {
x := 42
return &x
}store.go:4:22: moved to heap: xWith inlining disabled, the returned pointer must remain valid after Escape returns. The compiler places x on the heap.
Make a prediction to reveal the compiler’s result.
sync.Pool addresses another pattern: temporary objects with frequent reuse. A request might borrow a scratch buffer and return it when finished. Pooling can reduce allocation traffic, but the pool may discard an item at any time. It is not a bounded application cache, a durable ownership transfer, or a resource registry. Its per-P machinery is an implementation detail. A miss must always be correct, and keeping a huge backing array in a pooled buffer can retain far more memory than the next request needs. The Pool contract allows the runtime to drop unused items.
Do not use a finalizer to express “close this file when the request ends.” Finalizers are asynchronous, may delay reclamation, impose ordering complications, and need not run before process exit. Go 1.24 added runtime.AddCleanup and weak.Pointer. A cleanup passes a separate argument to a function after the watched object becomes unreachable; it avoids several finalizer restrictions and can support cycles. If the cleanup closure or its argument strongly references the watched object, it prevents its own trigger. Cleanup execution is still not deterministic. Use explicit Close or scope-based defer for timely resource release.
A weak.Pointer[T] lets you refer to an object without keeping it alive. Value() may return nil, so code must tolerate disappearance and rebuild or miss safely. It can also keep returning a non-nil value longer than you expect; collection timing is unspecified. Neither weak references nor cleanups are an eviction policy. runtime.KeepAlive can establish that a wrapper must remain reachable until a particular program point when an external resource is in use. Follow the cleanup rules and weak pointer contract rather than treating lexical scope as exact liveness.
The heap the collector inherits
Once our entry escapes, the collector needs more than its address. It needs the allocation’s size, its pointer layout and its mark state. The allocator creates that structure before the marker ever sees the object.
Go’s small-object allocator organizes heap memory into A contiguous run of Go heap pages, managed as fixed-size object slots for a small-object size class, or as one large allocation.. A Go heap page is 8 KiB in the cited runtime, regardless of whether Linux uses 4 KiB base pages. A small-object span belongs to one size class. If the class is 64 bytes, every object slot has the same stride. Knowing an interior pointer’s span and slot size lets the runtime recover the allocation’s base.
There are 67 nonzero small size classes in the Go 1.27.3 table, ending at 32,768 bytes. A 65-byte payload rounds to the 80-byte class; a 513-byte pointer-free payload rounds to 576. Above the small-object cutoff, the allocator uses whole-page large allocations. The generated table lives at internal/runtime/gc/sizeclasses.go, not the historical runtime/sizeclasses.go path.
Pay coordination costs only when refilling
The common allocation path starts with an mcache, associated with a P, the scheduler resource required to execute Go code. It caches spans with free slots for the different span classes. A bitmap cache lets allocation find a free slot efficiently. The cache follows the P as goroutines and OS threads are scheduled onto it. The Go concurrency dive explains why P, M and G are separate entities.
A free slot in a cached span. No shared heap lock on the usual fast path.
Refill from a span with free slots; sweep one if necessary.
Find whole Go pages, provision a new span and its metadata.
Grow the heap’s address space and make pages available.
When a cached span fills, mcentral supplies another span for that span class. When it needs fresh pages, mheap and the page allocator provide a run. Only after those runtime resources are insufficient does allocation need more memory from the OS. The hierarchy makes most allocations cheap without promising that every allocation has the same latency. Refill can include sweep work, and an allocating goroutine can also owe mark-assist work.
Size alone is not enough to select the span class. The runtime separates scan and noscan spans. Our Entry contains a pointer, so it belongs to a scan span. An equally large [64]byte needs no pointer tracing. A noscan object must still be kept alive when referenced, but once discovered it has no outgoing references for the marker to inspect. Noscan saves the work of tracing outgoing pointers; allocation and reachability bookkeeping still apply.
Type layout supplies a pointer map for scan objects. Small scan objects can use span-local pointer metadata; larger ones may need an allocation header referring to type information. This is why applying the bare size-class table to every Go type can give the wrong usable size or capacity. Pointerful and pointer-free allocations with the same requested bytes need not pay identical overhead.
Inspect a real class boundary
The visualizer below models pointer-free allocations on amd64 with Go 1.27.3 Green Tea enabled. It includes the 128 bytes reserved for inline mark/scan metadata on eligible small spans. A 64-byte noscan class therefore holds 126 slots in an 8 KiB span, not the table’s nominal 128 before runtime metadata. A scan span for our 64-byte entry additionally reserves 128 bytes for pointer bits, leaving 124 slots. Span capacity depends on the runtime version and collector configuration.
- Size class
- 6 · 64 B
- Span
- 8 KiB
- Objects / span
- 126
- Rounding / object
- 0 B
- Requested bytes
- Size-class rounding
Go 1.27.3 size-class table; full span occupancy on amd64 with Green Tea enabled. Shows up to 32 slots, equal width per slot. Pointer-free allocations avoid pointer maps and per-object scan headers. Small spans still reserve 128 B of Green Tea mark/scan metadata. Requests smaller than 16 bytes use a separate tiny-allocation path and are excluded.
Two kinds of waste appear even when every slot is occupied. Rounding waste is the difference between the requested bytes and class stride. Tail waste is leftover span space after fitting whole slots and reserved metadata. Real services also have partially occupied spans, per-P caches and pages stranded by scattered survivors. A tiny live object can prevent a page from becoming wholly free. A non-moving collector cannot gather those survivors into another span simply to release their original pages.
For pointer-free requests smaller than 16 bytes, the tiny allocator packs multiple values into a 16-byte block, subject to alignment. This saves allocation bookkeeping and slots, but couples reclamation: one live value can retain the whole block. That detail also matters for cleanups and finalizers on tiny objects. The visualizer excludes this path so a “slot” consistently means one allocation unit rather than a shared tiny block.
Finally, keep three maps distinct. Allocation bits record which slots are occupied. Mark bits record which objects were discovered in this collection. Pointer bits identify which words inside an object should be interpreted as references. Sweeping uses the first two to turn dead occupied slots into reusable ones. Marking uses the third to follow real pointers without treating a coincidental integer as an address. Green Tea adds explicit seen-versus-scanned bookkeeping, which we will use at the bottom of the dive.
The implementation boundary is visible in malloc.go, mheap.go and mbitmap.go. Together they answer the marker’s first question: given this address, which object and pointer layout am I looking at?
Allocation already organizes the heap into regular regions. The collector inherits those regions, their free slots, and their pointer metadata.
Keep the graph alive while it moves
Our cache now owns a collection of spans full of entries. To reclaim the dead entries, start by pretending the application is stopped. Put roots on a work list. Discover an object, record that it is live, and eventually inspect each of its pointer fields. Add newly discovered children to the work list. When the list is empty, every object reachable from those roots has been discovered. Anything still undiscovered can be swept.
The standard three-colour language describes that work. A white object is not discovered in this cycle. A grey object is discovered but has unprocessed outgoing references. A black object has had its references scanned. The colours are logical states, not literal bytes stored in every object header. A mark bit plus pending work can represent grey versus black; Green Tea makes the distinction more explicit for batched spans.
Root work includes live pointers in goroutine stacks, pointer-containing globals, and runtime-managed structures such as queued finalizer work. Finalizers need special handling: an unreachable object with a finalizer may be retained while the finalizer is scheduled, together with objects it must still access. Weak references are not ordinary strong root edges. The collector’s root machinery is richer than “scan every global and free everything else.”
Let the application run and the proof breaks
A simple sufficient invariant for a conventional incremental marker is: a black object must not point to an undiscovered white object. Without application writes, scanning preserves that condition. With writes, the application can install a fresh edge behind the marker’s back. The marker finishes honestly according to its work list and nevertheless reclaims something the program still needs.
Try the smallest useful failure. A and B are rooted. Initially B points to C, which points to D. Scan A first, while it has no child, making it black. Now move B’s reference to C into A and clear B’s reference before scanning B. A is no longer examined; B now has nothing to reveal. C and D remain white despite being reachable from A. Finish and sweep to see the bug.
A marker and a mutator share four objects
Roots point to A and B. Scan A, move B’s reference into A, then finish marking and sweep.
- Grey work queue
- A, B
- Lost live objects
- 0
Both roots are grey. C and D have not been discovered.
An algorithmic model: one pointer field per object, an explicit root scan, a FIFO queue and immediate shading. Go buffers barrier work and uses richer queues. The “move” is two ordered heap writes in one mutator.
The remedy is a write barrier: compiler-inserted work around relevant pointer writes. It records changed references so marking can remain correct while the application updates edges. Go 1.8 adopted a hybrid of a Yuasa deletion barrier and a Dijkstra insertion barrier, allowing it to avoid a full stack rescan at mark termination. The design is described in the hybrid barrier proposal and implemented in mbarrier.go.
The deletion half shades the old referent before overwriting a pointer slot. It prevents a mutator from hiding the last heap path by transferring a pointer to its stack. The insertion half shades the new referent while the current goroutine’s stack is still grey, preventing a hidden stack pointer from being installed into already-scanned heap storage. Here “shade” means discover if white and ensure its outgoing references will be processed.
writePointer(slot, new):
shade(*slot) // Yuasa deletion barrier
if current stack is grey:
shade(new) // Dijkstra insertion barrier
*slot = new
shade(p):
if p refers to a white heap object:
mark it grey
enqueue scan workThe proof uses the stack’s colour, while Go’s implementation conservatively records both old and new values at a pointer assignment. Buffering amortizes the work. The machine-level example later shows those two pointer values being placed into the write-barrier buffer before the store completes.
There is a subtle distinction between a teaching invariant and Go’s actual proof. A pure insertion barrier can maintain “no black-to-white edge.” Go’s hybrid design also admits cases justified by a weaker invariant: a white object remains protected by an appropriate path through grey work. A temporary black-to-white edge can therefore be safe when another path through grey work protects the target. The simulator uses immediate shading and a smaller graph to isolate the lost-edge problem.
Goroutine stack writes usually do not receive the same heap write barrier. The runtime coordinates scanning each stack with execution of that goroutine. Once scanned, the hybrid barrier’s guarantees keep stack-to-heap transfers safe without rescanning all stacks in one large pause. New heap objects allocated during marking are treated as black; zero-initialization and the barrier protocol protect subsequent pointer initialization. This avoids adding every new allocation to the ordinary “white then trace” path.
One cycle has several kinds of work
- Stop goroutines at safe points, complete required sweep work, enable the write barrier, and prepare marking. The barrier must be ready before mutators resume.
- Resume the application. Scan goroutine stacks with coordination, trace globals and other roots, and process discovered work. Background workers and allocation assists cooperate.
- When distributed mark work is exhausted, pause to complete mark termination and phase bookkeeping. No full second scan of every stack is required by the hybrid barrier design.
- Resume with the barrier off. Background sweeping and allocation-driven sweeping turn unmarked slots into free space. The next cycle must not start marking over unswept state.
Marking is distributed work. Per-P work buffers reduce shared contention, workers exchange batches, and the runtime must establish that no worker or barrier buffer can produce more unprocessed references before declaring the mark complete. An empty local queue is not enough. The phase transitions enforce that global agreement. They are the reason a concurrent collector still has brief stop-the-world intervals.
Background workers cannot always keep up with a burst of allocation. An allocating goroutine then performs a Collector work charged to an allocating goroutine so application allocation cannot indefinitely outrun marking progress.. Allocation creates debt in bytes; scanning repays it according to the pacer’s estimate of remaining work. This couples how fast the program can grow the heap to how fast it can finish finding live objects. It also explains why a service can suffer allocation latency while global pauses remain short.
Sweeping is separate from marking. After the live set is known, the runtime can inspect spans and make unmarked occupied slots free. A background sweeper does work, and allocation can sweep spans lazily when it needs capacity. A dead slot becoming free does not imply its whole span is empty. Nor does sweeping mean the scavenger has returned any physical memory to Linux. The GC phase description and sweep implementation keep these responsibilities separate.
Finish before the heap fills
We know how to find reachable entries without stopping every request for the entire trace. The next question is when to start. Starting too late lets allocation outrun the collector and forces expensive assists. Starting too early repeats the same graph walk unnecessarily. The pacer estimates the relationship between allocation progress, scan work and available CPU so a cycle can finish near its heap goal.
The percentage-based goal has a useful high-level equation. Let L be the live heap after the preceding collection, and R the scannable roots: stacks and globals. With a non-negative percentage setting G, the target is approximately:
Heap goal = L + (L + R) × GOGC / 100
For our service, suppose L is 64 MiB and R is 8 MiB. At GOGC=100, the target is 64 + 72 = 136 MiB. The extra 72 MiB is allocation headroom, not another copy of the live graph. At GOGC=200 the target becomes 208 MiB. At GOGC=50 it becomes 100 MiB. Larger headroom means more allocations can occur between collections of roughly the same live graph.
Including roots matters for services with many goroutines. If a substantial fraction of the work lives in stacks, a policy based only on live heap underestimates the cost of a cycle. Go 1.18’s pacer redesign included root work in the policy. In this equation, R scales the growth allowance and L remains the heap baseline; stack storage is accounted for separately. The official GC guide gives this equation; the pacer redesign explains why it changed.
The goal is not the trigger. A concurrent marker needs a runway. The pacer uses observed scan throughput and allocation behaviour to start before the goal, then adjusts assist ratios as estimates change. Runtime minimums, uncertainty margins and bursts make the equation an estimate rather than an exact heap ceiling. Allocating one huge object can cross a target in a single step. A new workload can change both the live set and scan cost during the cycle.
A memory limit spends that runway
With a finite memory limit, the runtime derives another heap goal from the budget remaining after non-heap runtime memory and accounting adjustments. It takes the more restrictive policy, subject to safety floors and reserves. The implementation also accounts for free heap space, released pages and internal headroom; subtracting a fixed “other” number is only a teaching approximation. The limit is measured roughly as MemStats.Sys - MemStats.HeapReleased, or the equivalent total-memory metric minus released heap bytes.
Suppose the service’s non-heap runtime memory consumes 24 MiB of a 160 MiB limit. That leaves about 136 MiB for the heap in the simple model, exactly our percentage goal. Tighten the limit and collection must run more frequently. Set it below the live heap plus unavoidable runtime memory and there is no successful collection that could satisfy it. The collector cannot free reachable cache entries just because the budget demands it.
A steady live heap, a finite CPU budget
- GOGC goal
- 136 MiB
- Constrained goal
- 136 MiB
- Model GC CPU
- 11.1%
- Cycle interval
- 2.53 s
- Heap allocations
- Constrained heap goal
- Limit minus non-heap memory
Read the model’s assumptions
One CPU-equivalent resource; 8 MiB of scannable roots, 24 MiB of non-heap runtime memory, and 256 MiB of scan work per GC CPU-second. Each cycle costs W = (live + roots) / 256 seconds. A headroom of H takes H / allocation mutator seconds to fill. GC CPU share is W / (W + H / allocation). If that exceeds 50%, this model allows H to grow to allocation × W. Sweeping is drawn as an instantaneous drop. Real Go overlaps marking, allocation and sweeping, triggers before the goal, adds internal reserves, and enforces its limiter over a time window. This is a time/space model, not a runtime forecast.
That impossible configuration is the death spiral: collect, discover the same live graph, allocate almost nothing, immediately collect again. A collector willing to take all CPU could leave the application unable to do the work that would release its references. Go’s memory-limit machinery therefore includes a GC CPU limiter. The documented policy is roughly 50% of available CPU over a window of 2 × GOMAXPROCS CPU-seconds, with allowance for short transients. The window lets the runtime absorb short spikes while preserving application progress.
When the guard limits collection work, the heap may exceed the soft memory limit. That choice can preserve application progress while increasing the risk of hitting an external hard limit. An undersized container can still hit its hard limit. Queuing, cache interference, allocation bursts and kernel pressure can also amplify request latency beyond the direct CPU cost. Read limiter activity alongside assist CPU and request latency. The metric /gc/limiter/last-enabled:gc-cycle reports the latest cycle in which the limiter was enabled.
Keep the 50% guard separate from the ordinary 25% background-worker target. During marking, the runtime aims to devote roughly one quarter of GOMAXPROCS capacity to background mark workers. Idle workers can use spare capacity, and assists can add collector CPU when allocation outruns progress. The 25% figure is a scheduling target during a phase, not “GC always consumes a quarter of your process” and not a maximum. Between cycles there is no concurrent mark workload to schedule.
The practical experiment is controlled: hold workload and container resources steady, record allocation rate, live heap, goal, scan bytes, assist CPU and latency, then change one policy. If a larger goal cuts CPU but pushes the process into memory pressure, you have moved the trade-off rather than fixed an inefficiency. If live heap steadily rises, examine retention before increasing the budget. The exact controller is in mgcpacer.go, and the progress guard is in mgclimit.go.
Free to Go is not free to the OS
We unlink a cache chain, a collection finds it unreachable, and sweeping frees the slots. Yet the container’s memory graph hardly moves. This is not automatically a leak. The allocator has gained reusable capacity, while the OS may still consider the backing pages resident. Object reachability, allocator occupancy, mapped address space and physical residency are four different layers of the same memory.
The runtime gets virtual address space in larger regions and tracks heap metadata through arenas. On linux/amd64 in the cited implementation, the heap-arena granularity is 64 MiB. An arena here is a runtime addressing and metadata unit, not a promise that 64 MiB of RAM is immediately committed, and not the separate experimental user-arena API. The runtime manages many 8 KiB Go pages inside that address space and combines them into spans.
Linux mmap reserves or maps virtual ranges; touching pages can require the kernel to supply physical storage. Go’s mapping lifecycle includes reservation, preparation and use states. The exact flags differ across platforms, and a large virtual size alone says little about the resident cost. The virtual-memory level in the memory dive follows demand paging in depth. Here the important consequence is that returning physical backing does not require giving up every reserved virtual address.
Still reachable
Sweep → reusable
Scavenge → released
The scavenger works in pages
Sweeping reclaims object slots. The scavenger releases backing for free heap pages so the OS can use that memory elsewhere. Background scavenging is paced, and allocation can induce additional scavenging under memory pressure. Partially occupied pages cannot simply be discarded: the live bytes still matter. A cache that leaves one survivor on many pages can have modest live payload and a much larger retained footprint.
Linux has two relevant advice mechanisms. MADV_DONTNEED discards private anonymous page contents; later accesses obtain zero-filled memory, and RSS normally reflects the release promptly. MADV_FREE marks contents as lazily reclaimable. Until the kernel reclaims them or they are dirtied again, pages may remain resident and old contents may still be readable. That difference can make a process look large in an RSS graph even after it has offered memory back.
Go switched toward MADV_FREE in Go 1.12, then made MADV_DONTNEED the Linux default again in Go 1.16 so accounting better matched operator expectations. In Go 1.27.3, the default remains DONTNEED; GODEBUG=madvdontneed=0 selects the alternative where supported. Other operating systems use different memory-release mechanisms. The Go 1.16 runtime notes, Linux memory implementation and scavenger explain the division of responsibilities.
runtime.GC() forces a collection and waits for it to complete. RSS can remain above live heap because scattered survivors still occupy pages and the collector leaves heap objects in place. debug.FreeOSMemory() requests a collection and an attempt to return memory to the OS. It can be useful in special phases, but putting it in a request loop usually replaces amortized runtime policy with repeated global work.
The container has the last word
A cgroup hard memory limit is enforced outside the runtime. It can account for memory the Go heap budget does not capture, including native allocations, mapped files and other charged resources. RSS also includes thread stacks, code and resident mappings; shared pages complicate ownership. A database library called through cgo can grow while Go’s managed heap stays steady. A working-set graph, RSS graph and /gc/heap/live:bytes graph therefore need not agree.
Set GOMEMLIMIT with headroom for the whole deployment, measured under realistic bursts. A percentage below the cgroup ceiling is a starting hypothesis, not a portable constant. The official GC guide suggests allowing additional room for untracked sources; larger native heaps or unusual mappings need more careful accounting. In Go 1.25 and later, container-aware GOMAXPROCS can adapt CPU capacity to cgroup constraints. That helps worker scheduling, but it does not turn GOMEMLIMIT into an automatically chosen container memory limit.
Our cache example gives a useful diagnostic sequence. If live heap falls but allocated heap falls slowly, inspect sweep progress. If allocated heap falls but retained runtime memory stays high, inspect page occupancy and released-page metrics. If those improve while RSS stays high, investigate OS advice, native memory and mappings. If live heap never falls, look for a strong reference path. These observations separate retention from allocator fragmentation and kernel residency without requiring a new knob for each symptom.
A pointer write becomes instructions
The barrier looked like a tiny algorithm when written as pseudocode. What does inserting an entry actually make the CPU execute? Compile the function Store(slot **int, p *int) { *slot = p } with go tool compile -S store.go. The excerpt below was produced by Go 1.27.3 for linux/amd64. It uses Go assembler syntax, and omits the stack check, prologue and metadata directives.
// func Store(slot **int, p *int) { *slot = p }
// AX = slot; BX = p. Go 1.27.3, linux/amd64.
// Excerpt; stack prologue, metadata and slow stack path omitted.
0x000a CMPL runtime.writeBarrier(SB), $0
0x0011 JEQ 34
0x0013 MOVQ (AX), CX
0x0016 CALL runtime.gcWriteBarrier2(SB)
0x001b MOVQ BX, (R11)
0x001e MOVQ CX, 8(R11)
0x0022 MOVQ BX, (AX)Under the internal calling convention here, AX holds the destination slot address and BX holds the new pointer. The CMPL checks runtime.writeBarrier. When it is zero, JEQ 34 jumps to decimal offset 34, hexadecimal 0x22, and MOVQ BX, (AX) performs the assignment. The application still pays a barrier-state check at this site outside marking, but skips the recording path.
When the barrier is enabled, MOVQ (AX), CX loads the old pointer. runtime.gcWriteBarrier2 reserves space for two pointer words and returns that buffer address in R11. The next two stores record the new pointer and the overwritten pointer; only then does the final store update the actual object graph. A full buffer takes a slower path to flush and process entries. The write barrier does not recursively mark an entire subgraph at every assignment.
The buffer belongs to the executing P. Amortizing its flush makes the fast path much cheaper than a general runtime call per pointer. The compiler can also elide barriers for writes it proves safe, such as some writes into stack storage. Bulk copies of pointer-containing values require corresponding barrier handling. Integers do not receive a heap pointer barrier merely because their bit pattern resembles an address. Relevant implementations include mwbbuf.go and the amd64 barrier assembly.
The compiler and runtime also coordinate safe points. A collection phase transition must not interpret execution as safely paused in the middle of a critical barrier sequence. Compiler metadata marks regions where asynchronous preemption cannot simply stop and scan arbitrary intermediate state. This is one place where the barrier and the preemption machinery meet: a short sequence that appears local in disassembly participates in a whole-runtime protocol.
How does the marker know a word is a pointer?
Our entry’s first word is next, an actual pointer. The next word is an integer key; the remaining six words are payload bytes. The marker must follow only the first word. Scanning every word as a possible address would produce accidental retention, complicate correctness, and make movement or precise stack handling harder. Go instead carries pointer-layout information from the compiler into the runtime.
Entry · eight 8-byte words on amd64
Logical word map, not a dump of the runtime’s packed metadata. Physical encoding and placement depend on object size and collector. Scalar bytes that resemble addresses remain scalar.
For heap objects, type information and allocator metadata describe pointer-bearing positions. Arrays repeat element layouts; structs compose field layouts and padding. In Go 1.27.3, small type masks are emitted by the compiler; larger type masks can be built on demand by the runtime instead of embedding every expanded bitmap in the binary. This is described by GCData and TFlagGCMaskOnDemand. The location of pointer metadata also depends on allocation size and path: some allocations use span metadata, while others carry a header.
“GC shape” is another use of this layout idea. Types can share relevant runtime structure even when their source-level names differ. In generic compilation, shape-based code sharing groups compatible representations; pointer layout is one relevant constraint, though generic shape rules are not simply “same size means same shape.” For the collector, the crucial property is an accurate description of which words can contain pointers, not the spelling of the original type.
Stack frames need liveness information as well as types. A local can be out of scope or dead before a function returns, and a physical stack slot can be reused for another value. At safe points, the program counter selects compiler-generated stack maps identifying live pointer slots in the frame. The runtime walks frames using unwind information and scans the locations described by those maps. Register-based calling conventions add coordination to make live values visible in the required form; “scan a stack” does not mean blindly reading every register and byte as a pointer.
This precision explains two rules that often surprise Go programmers. First, a receiver may become unreachable after its last actual use, before the function text ends, which is why resource wrappers sometimes need KeepAlive. Second, converting an address to uintptr does not keep the pointee alive. It is an integer, not a GC reference. The permitted unsafe.Pointer conversions have specific rules; storing a numeric address for later use can bypass the lifetime information the collector relies on.
At this layer, our original cache entry is no longer an abstract object. It is an address resolved to a span, a slot with allocation and mark state, and eight words with a known pointer pattern. The marker can now load the next pointer with precision. What remains is the physical cost of reaching those words.
Mark in the order memory prefers
Imagine a large cache chain whose entries were allocated at different times on different Ps. Following next can jump across spans and across virtual pages. The marker knows exactly what to load, but the processor may have to wait for it. A dependent pointer load cannot reveal its successor’s address until the current pointer arrives. More cores help only while there is independent work and enough bandwidth to feed them.
This is why marking is often limited by the memory system rather than arithmetic. The useful operation can be “load one pointer, test metadata, discover another object.” The bytes arrive in cache lines, often 64 bytes on the x86 systems discussed here, while the code may use only a small part of each fetched line. Misses in address-translation caches add page-table work. Larger live graphs stress cache capacity and the TLB, and unrelated application traffic competes for the same hierarchy.
The cost of each pointer load depends on topology, cache residency, memory-level parallelism, page size and prefetching. The TLB, cache-line and DRAM levels of the memory dive explain those costs. Our collector-specific question is narrower: can the runtime choose a scan order that makes better use of memory it already fetched?
Green Tea changes the work unit
A traditional object work list encourages a graph flood: discover an object, later scan it, discover more objects elsewhere. It can produce good locality by accident but does not organize the work around adjacent allocations. Green Tea batches eligible small-object work by span. Discovering an object records it as seen and schedules its span. Later, a worker scans the accumulated pending objects in that span, giving nearby discoveries time to join the same batch.
Our entry stays at the same address. Green Tea groups pending scans using Go’s allocator pages and spans, with specialized paths for eligible small objects and separate handling for larger objects. If another reference later discovers a previously unseen object in an already-processed span, that span can need more work. A single pass over memory would not be enough.
Same discovered objects. Different scan order.
- Objects scanned
- 0 / 16
- Line misses
- 0
- Resident lines · oldest first
- empty
Step through the memory accesses. Only scanned objects bring a line into this toy cache.
An isolated, already-discovered grey frontier, not a full Green Tea implementation. Objects are 16 bytes; cache lines are 64 bytes; two toy spans are 128 bytes each (real Go pages are 8 KiB). Fully associative LRU, no prefetching, no metadata accesses. Both orders touch four unique lines; only reloads differ.
The comparison starts with a grey frontier: objects that have already been discovered. It isolates access order from the harder question of which references have already been discovered. Object order visits four lines repeatedly; batching visits neighbours while their line is resident. With a cache large enough to hold all four lines, both orders miss only four times. The benefit depends on which lines fit in the cache and how often the scan order revisits them.
Green Tea separates “seen” from “scanned” bits so a batch can find discovered objects whose outgoing pointers are still pending. The conceptual operation below computes an active-object mask, expands it according to object size, and intersects it with the pointer map. Scalar words and already-processed objects disappear from the scan work. These are logical masks with bit positions shown left to right, not a dump of a particular machine-endian structure.
seen = 00101100
scanned = 00000100
active = seen & ^scanned = 00101000
newScanned = seen | scanned = 00101100
// Expand active objects to their words.
// Then keep only pointer-bearing words:
activePointers = expand(active) & pointerMapRegularity also changes what instructions are useful. The Go 1.26 release added vectorized scanning on suitable amd64 processors. The fast path needs AVX-512 plus GFNI and other features. Go 1.27.3’s feature gate checks AVX512VL, AVX512BW, GFNI, AVX512BITALG, AVX512DQ, AVX512VBMI and POPCNT.
The vector kernel manipulates masks and collects pointer words from regular regions. Following pointers to other pages still incurs memory-access costs. Go’s Green Tea design article describes the use of GFNI’s VGF2P8AFFINEQB for expanding object masks, along with wide pointer scanning. Without the required hardware features, the collector still uses its non-vector paths. The locality design remains valuable independently of that fast kernel.
From experiment to default
| Version | Verified status | What to infer |
|---|---|---|
| Go 1.25 | Green Tea available through GOEXPERIMENT=greenteagc at build time. | An experiment to measure, not that release’s default. |
| Go 1.26 | Green Tea enabled by default; eligible amd64 vector scanning added. | Rebuilding can change collector performance without source changes. |
| Go 1.27.3 | Installed source defaults GreenTeaGC to true and contains span-batched marking and hardware-gated vector scanning. | The implementation examined here. Do not infer every workload benefits equally. |
The Go 1.25 notes document the experiment; the Go 1.26 notes document the default and vector acceleration. For 1.27.3, the claim above is checked directly against the default experiment configuration, span-based marking and the feature gate. These source checks establish the configuration and scanning paths used for this dive.
The Go team’s benchmark improvements depend on the workload. If only a small fraction of your CPU was GC, even a large collector improvement makes a modest difference to total CPU. If your cache is pointer-free, already cache-resident, or dominated by another bottleneck, the profile can look very different. Measure total throughput, assist CPU, collector CPU and memory under the same workload and policy.
The entry’s whole path now fits together. A request publishes a pointer to a 64-byte entry. The allocator chooses a scan span and installs layout information. A paced concurrent marker preserves the reachable graph while barriers record mutations. Sweeping makes dead slots reusable; scavenging may later release free pages. The compiler makes those guarantees concrete with checks, buffers and stack maps. Finally, the order in which marking touches memory determines how much useful work each fetched cache line can support.
For our cache service, that leaves a practical order of attack: bound what stays reachable, reduce needless allocation and pointer-rich structure, provide a realistic memory budget, and then measure policy changes. GOGC chooses how much runway to buy. Green Tea tries to spend fewer stalled cycles traversing what remains. Neither can make a permanently rooted cache entry become garbage.
A collector traces a graph, but a processor fetches memory in lines. Green Tea improves the fit between those two units without changing which objects must survive.
End of Go garbage collection · 8 levels, Idea to Silicon
Where to next
Next in Languages and runtimes · 04 of 06
Rust ownership: the checks and the machine
Follow a batch of log lines through moves, borrow checking, threads and async suspension. See what the compiler proves, what a future stores, and the instructions that remain.
Practical notes
Tune Go GC with GOGC and GOMEMLIMIT
Use GOGC to trade heap growth for collection frequency and GOMEMLIMIT to constrain Go-managed memory. Set container headroom explicitly: Go 1.27.3 does not derive this limit from cgroups.