Languages and runtimesDive 02 of 06

Go concurrency, from go keyword to OS thread

What makes goroutines and channels different, the patterns they unlock and the bugs they invite, then down through the G-M-P scheduler, the inside of a channel, OS threads and the netpoller, to the two instructions that make stacks growable.

Levels
8
Reading
45 min
Depth
Idea to Machine
Published
Ggoroutines10,000 runnable or parked
PprocessorsGOMAXPROCS = 4
MOS threadsone per running P
CPUcoresthe hardware

Level 01/08Idea

Concurrency as a language feature

You are writing a link checker. It has a list of 1,000 URLs, and for each one it sends a HEAD request and records the status code. Each request takes about 200 ms, and almost all of that is waiting: for DNS, for the TCP handshake, for a server on another continent to answer. The CPU work per URL is a few microseconds.

Written the obvious way, the program does one thing at a time:

checklinks.go, version 1
for _, u := range urls {
    status, err := check(u) // ~200 ms, almost all of it waiting
    report(u, status, err)
}

That takes 1,000 × 200 ms, a bit over three minutes, and the machine is idle for nearly all of it. The fix is not more CPU. It is to stop waiting for one request before starting the next: have hundreds of requests in flight, and handle each answer when it arrives. That is concurrency.

Concurrency is not parallelism

Rob Pike's phrasing is the one that stuck: concurrency is about dealing with lots of things at once, parallelism is about doing lots of things at once. Concurrency is a property of how a program is structured: independent tasks that can make progress without waiting for each other. Parallelism is a property of how it runs: several of those tasks executing on different cores in the same instant. A concurrent program can run on one core. A parallel one needs several.

Try both kinds of workload below. The URL fetches are mostly waiting; the file hashes are pure computation.

Workload
Run them

Each bar is one task over time (1 unit = 50 ms). Solid segments occupy a core, hatched segments wait on the network, thin red lines are tasks that are ready but have no core. The vertical line marks when the last task finishes.

For the link checker, concurrency on a single core already gets most of the win, because the expensive part is waiting and waits can overlap. For hashing, only extra cores help. Real programs are a mix, and Go's runtime gives you both from the same code: you write concurrent structure, and it runs in parallel on as many cores as it is allowed to use.

Why it had to be a language feature

Every mainstream language can do concurrent I/O. The question is what it costs you. There are two classic answers, and each has a price.

  • One OS thread per task. You write plain blocking code, and the kernel switches between threads. But threads are heavy: each reserves a large stack up front (8 MB of address space by default on Linux), creating one is a system call, and every switch goes through the kernel. Ten thousand threads is painful; a million is out of the question.
  • An event loop with callbacks, promises or async/await. One thread juggles thousands of tasks, each of which is just a small heap object. This is cheap, but it splits your code into two worlds. An async function can only be awaited from another async function, so the property spreads upward through every caller. Bob Nystrom called this the From the essay 'What color is your function?' (2015): async and sync functions are two 'colors' that cannot freely call each other, so the async color spreads through a codebase. problem.

Go took a third path. It made the thread cheap instead. A goroutine is a function running independently, scheduled by the Go runtime rather than the kernel, starting with a stack of a couple of kilobytes that grows only if it needs to. Blocking a goroutine costs about as much as an event loop parking a callback, so you can write plain blocking code, one goroutine per task, and the runtime turns it into an event loop underneath. There is no async keyword and no function colors. Any function can be run concurrently by putting go in front of the call:

checklinks.go, version 2 (incomplete)
for _, u := range urls {
    go check(u) // starts a goroutine and returns immediately
}

That loop finishes in microseconds and starts 1,000 goroutines. It is not a correct program yet: nothing waits for them, and nothing collects their results. Those two problems, waiting and communicating, are what the rest of Go's concurrency vocabulary is for.

The CSP heritage

Go's answer to communication comes from Tony Hoare's 1978 paper Communicating Sequential Processes: independent sequential processes that share nothing and interact only by sending messages over channels. The idea ran through a line of Bell Labs languages for two decades before Go: Squeak and Newsqueak (Rob Pike, the first with Luca Cardelli), Alef (Phil Winterbottom) and Limbo (Sean Dorward, Pike and Winterbottom). Go's goroutines, channels and select statement descend directly from those.

The slogan from Effective Go sums up the intended style:

Do not communicate by sharing memory; instead, share memory by communicating.

In practice Go is less pure than CSP. Goroutines share one address space, and the standard library has mutexes and atomics, which are often the simpler tool for guarding a counter or a cache. The slogan is a default, not a law: when data moves between goroutines, pass ownership of it through a channel rather than having both sides poke at it.

How Go compares

Every row in this table is some language's answer to "how do I run many tasks without many kernel threads?" The interesting columns are where the stack lives and when a task can be interrupted.

UnitScheduled byStack modelCost to startCan be interruptedColored functions
OS thread (C, Java platform threads, Rust std::thread)KernelFixed-size stack reserved up front, committed as touchedA syscall; MBs of address space, tens of KB realAnywhere (timer interrupt)No
GoroutineGo runtime, M:N onto threadsContiguous, growable, starts at 2 KB, copied when it growsA few hundred ns, ~2.5 KBAt function calls; anywhere (via signals) since Go 1.14No
JavaScript async/awaitOne event loop threadStackless: a promise plus a closureAn object allocationOnly at awaitYes
Rust async (e.g. Tokio)Library executor, M:NStackless state machine, size known at compile timeA heap allocation of the future's sizeOnly at .awaitYes
Python asyncioOne event loop threadStackless coroutine objectAn object allocationOnly at awaitYes
Java virtual threads (JDK 21+)JVM, M:N onto carrier threadsFrames copied to heap chunks when the thread unmountsHundreds of bytes, plus heap stack chunks as it growsAt blocking calls only, no time slicingNo
Erlang processBEAM, one scheduler per corePrivate heap and stack, 338 words (~2.7 KB) at spawnAround a microsecondAt any function call, once its reduction budget is spent (loops are recursion, so code can't avoid it)No

Two neighbours are worth singling out. Erlang is the closest relative in spirit, but its processes share no memory at all: every message is copied into the receiver's heap, so data races are impossible by construction. Go chose shared memory plus a race detector. Java's virtual threads arrived more than a decade after goroutines with the same pitch (plain blocking code, cheap threads, no colors), which is a fair sign that the idea held up.

Check yourselfYour program hashes 4 large files, and Go is limited to using one core. You launch one goroutine per file. Compared with hashing them in a plain loop, what happens?

Level 02/08Interface

Goroutines, channels and select

Back to the link checker. It needs three things: a way to start work (go), a way to wait for it, and a way to get results back. Go gives you a small, composable set of tools for these. Learn their exact semantics, because nearly every concurrency bug in Go is a misunderstanding of one of them.

Goroutines

go f(x) evaluates f and x right away in the current goroutine, then runs the call in a new one. You get nothing back: no handle, no future, no ID, no way to kill it from outside. That is deliberate. A goroutine ends when its function returns, and when main returns the whole program exits, taking every other goroutine with it mid-sentence. So "how will this goroutine stop, and who waits for it?" is a question you have to answer for every go statement you write.

The simplest answer to "who waits" is a sync.WaitGroup, a counter you can block on until it reaches zero:

waiting for a batch
var wg sync.WaitGroup
for _, u := range urls {
    wg.Add(1)
    go func() {
        defer wg.Done()
        check(u)
    }()
}
wg.Wait() // blocks until the counter is back to zero

// Go 1.25 and later: the same thing, minus the bookkeeping
for _, u := range urls {
    wg.Go(func() { check(u) })
}
wg.Wait()

Channels

A channel is a typed, goroutine-safe conduit. ch <- v sends, v := <-ch receives, and v, ok := <-ch also tells you whether the value came from a real send (ok is false once the channel is closed and empty). for v := range ch receives until the channel is closed.

The capacity you pass to make changes the semantics, not just the performance. An unbuffered channel, make(chan T), has no storage: a send blocks until a receiver takes the value, so the two goroutines meet at that point. It is a synchronisation as much as a data transfer. A buffered channel, make(chan T, n), is a queue of up to n values: sends block only when it is full, receives only when it is empty.

checklinks.go, version 3
results := make(chan Result) // unbuffered

for _, u := range urls {
    go func() {
        status, err := check(u)
        results <- Result{u, status, err} // blocks until someone receives
    }()
}

for range urls {
    r := <-results // blocks until someone sends
    report(r)
}

Here is a channel you can drive directly. Every outcome is computed from the same rules the runtime follows, so try to predict each one before you click.

Make the channel
Then

Blocked senders (sendq)

    ch

    Blocked receivers (recvq)

      What happened

        A channel playground. Each button starts a new goroutine (G2, G3, …) that performs one operation. Try: an unbuffered send with no receiver; three sends into the buffered channel then a fourth; close with goroutines still blocked; close twice. In a real program an unrecovered panic ends the whole process; here the playground carries on.

        The rules fit in one small table. It is worth memorising, since the two panics and the two eternal blocks in it are the source of most channel bugs.

        Operationnil channelopen channelclosed channel
        ch <- vblocks foreverblocks until a receiver or buffer spacepanics
        <-chblocks foreverblocks until a value is availablebuffered values first, then the zero value with ok == false, immediately, forever
        close(ch)panicswakes all waiters; later receives drain then return zeropanics

        The closed-channel row hides Go's most useful trick. A receive on a closed channel never blocks, for any number of receivers. So close is a broadcast: one call wakes every goroutine waiting on that channel, now and in the future. A send can only ever wake one. Cancellation in Go is built on exactly this.

        And the nil row is a feature too: inside a select, setting a channel variable to nil disables that case, which is a clean way to stop reading from an input once it is exhausted.

        select

        select is a switch over channel operations. It blocks until one of its cases can proceed, then runs that one. If several are ready at once, it picks one uniformly at random, so no case can starve the others. With a default case it never blocks at all, which makes non-blocking sends and receives possible.

        waiting on several things at once
        select {
        case r := <-results:
            report(r)
        case <-time.After(2 * time.Second):
            log.Println("still waiting after 2 s")
        case <-ctx.Done():
            return ctx.Err() // context.Canceled or context.DeadlineExceeded
        }

        context: cancellation that crosses API boundaries

        ctx.Done() in that last case is a channel that gets closed when the work should stop: the user hit Ctrl-C, the HTTP client went away, a deadline passed, or a sibling failed. A context.Context is passed as the first argument down the call chain, and contexts form a tree: cancelling a parent closes the Done channel of every child derived from it. It is the broadcast-by-close trick, packaged as an API, and the parts of the standard library that wait on the network (HTTP clients and servers, net.Dialer, database/sql) honour it.

        a check that can be cancelled
        func check(ctx context.Context, u string) (int, error) {
            req, err := http.NewRequestWithContext(ctx, "HEAD", u, nil)
            if err != nil {
                return 0, err
            }
            resp, err := http.DefaultClient.Do(req) // aborts when ctx is done
            if err != nil {
                return 0, err
            }
            resp.Body.Close()
            return resp.StatusCode, nil
        }
        
        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()

        Mutexes are fine

        Not everything should be a channel. Counting status codes in a shared map is a job for sync.Mutex: lock, update, unlock, done. A useful rule of thumb is that channels are for transferring ownership of data and coordinating goroutines, and mutexes are for guarding state that several goroutines touch in place. sync/atomic covers single counters and flags, and sync.Once runs initialisation exactly once.

        shared state, guarded
        type Stats struct {
            mu     sync.Mutex
            byCode map[int]int
        }
        
        func (s *Stats) Add(code int) {
            s.mu.Lock()
            defer s.mu.Unlock()
            s.byCode[code]++
        }

        errgroup: the batteries-included version

        Most real fan-out code wants all of the above at once: run N things, limit how many run at a time, stop everything on the first error, and return that error. golang.org/x/sync/errgroup does exactly that, and it is the right default for the link checker:

        checklinks.go, version 4
        import "golang.org/x/sync/errgroup"
        
        g, ctx := errgroup.WithContext(ctx)
        g.SetLimit(16) // at most 16 checks in flight
        for _, u := range urls {
            g.Go(func() error {
                _, err := check(ctx, u)
                return err // the first error cancels ctx for everyone else
            })
        }
        if err := g.Wait(); err != nil {
            return err
        }
        Check yourselfA goroutine runs `v, ok := <-ch` on a channel that has been closed and still holds two buffered values. What does it get?

        Level 03/08Interface

        Patterns they open

        Those few primitives compose into a handful of patterns that cover most real concurrent code. They all answer the same three questions: who produces the work, who consumes it, and how does everyone learn it is time to stop?

        Pipelines

        A pipeline is a chain of stages connected by channels. Each stage is a goroutine (or several) that receives from an inbound channel, does something, and sends to an outbound one. The stopping rule is simple and important: the stage that sends on a channel is the one that closes it, and it closes it when it is done. The next stage's range loop then ends on its own, closes its own output, and the shutdown ripples down the chain.

        a three-stage pipeline (after the Go blog's 'Pipelines and cancellation')
        func gen(nums ...int) <-chan int {
            out := make(chan int)
            go func() {
                defer close(out) // closing tells the next stage "no more"
                for _, n := range nums {
                    out <- n
                }
            }()
            return out
        }
        
        func sq(in <-chan int) <-chan int {
            out := make(chan int)
            go func() {
                defer close(out)
                for n := range in { // ends when in is closed and drained
                    out <- n * n
                }
            }()
            return out
        }
        
        for n := range sq(sq(gen(1, 2, 3))) {
            fmt.Println(n) // 1, 16, 81
        }

        Fan-out and fan-in

        Fan-out is several goroutines receiving from the same channel. Each value goes to exactly one of them, so they share the load with no extra coordination. Fan-in is the reverse: several channels merged into one. The merge needs a WaitGroup, because its output can only be closed once every input is drained, and only one goroutine may close it.

        gensqsqsqmerge
        Fan-out then fan-in. Three sq stages read the same input channel, so the work spreads across them; merge forwards whatever arrives first, so output order is not guaranteed.
        merge
        func merge(cs ...<-chan int) <-chan int {
            var wg sync.WaitGroup
            out := make(chan int)
            for _, c := range cs {
                wg.Go(func() {
                    for n := range c {
                        out <- n
                    }
                })
            }
            go func() {
                wg.Wait()  // every input drained
                close(out) // only now is it safe to close
            }()
            return out
        }
        
        in := gen(1, 2, 3, 4, 5, 6)
        // fan-out: three workers read the same channel
        c1, c2, c3 := sq(in), sq(in), sq(in)
        // fan-in: one channel collects all of them
        for n := range merge(c1, c2, c3) {
            fmt.Println(n) // order is not guaranteed
        }

        Worker pools

        One goroutine per URL is fine for 1,000 URLs. For 10 million, or when each task holds a database connection, you want a fixed number of workers pulling from a shared jobs channel. Because a free worker grabs the next job as soon as it is idle, the pool balances itself: slow jobs do not hold up fast ones. Change the number of workers and see where adding more stops paying off.

        Wall time
        Finished
        Cancelled mid-flight
        Never started
        Twelve jobs of different lengths (100 to 450 ms) on a worker pool, computed live: each job goes to whichever worker frees up first, which is what a shared jobs channel does. With the deadline on, in-flight jobs are cut off at 600 ms and the rest are never started.

        With one worker you get the serial time, 2,500 ms. Going from one to four workers cuts it by roughly four. From about seven workers on, the chart stops improving: the 450 ms job (number 6) can't be split, so it alone sets the floor. More workers can't make the longest single job shorter.

        Bounded parallelism with a semaphore

        Sometimes you want one goroutine per task, for simplicity, but at most N running at once, so you don't open 10,000 sockets to the same host. A buffered channel of empty structs is a counting semaphore: sending acquires a slot and blocks when all N are taken; receiving releases one. errgroup.SetLimit does the same thing internally, and golang.org/x/sync/semaphore adds weights.

        a pool that respects a deadline
        jobs := make(chan string)
        results := make(chan Result)
        
        var wg sync.WaitGroup
        for range workers { // Go 1.22 range-over-int
            wg.Go(func() {
                for u := range jobs { // each worker pulls the next free job
                    status, err := check(ctx, u)
                    results <- Result{u, status, err}
                }
            })
        }
        
        go func() {
            defer close(jobs)
            for _, u := range urls {
                select {
                case jobs <- u:
                case <-ctx.Done():
                    return // stop handing out work once the deadline passes
                }
            }
        }()
        
        go func() {
            wg.Wait()
            close(results)
        }()
        
        for r := range results {
            report(r)
        }
        a channel as a counting semaphore
        sem := make(chan struct{}, 16) // capacity = max in flight
        
        var wg sync.WaitGroup
        for _, u := range urls {
            sem <- struct{}{} // acquire: blocks while 16 are running
            wg.Go(func() {
                defer func() { <-sem }() // release
                check(ctx, u)
            })
        }
        wg.Wait()
        ask every mirror, take the fastest, cancel the rest
        func fastest(ctx context.Context, mirrors []string) (string, error) {
            ctx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
            defer cancel() // tells the losers to give up
        
            ch := make(chan string, len(mirrors)) // room for every answer
            for _, m := range mirrors {
                go func() {
                    if _, err := check(ctx, m); err == nil {
                        ch <- m // never blocks, even if nobody reads it
                    }
                }()
            }
            select {
            case m := <-ch:
                return m, nil
            case <-ctx.Done():
                return "", ctx.Err()
            }
        }

        Timeouts and cancellation

        Every pattern above has a hidden fourth question: what if the consumer goes away? A pool that keeps sending results nobody reads, or a stage blocked forever on a full channel, is a leak. The defence is the same everywhere: every blocking send or receive in a long-lived goroutine sits in a select with a case <-ctx.Done(), and every call that can block takes the context. Look at the worker pool tab: the producer stops handing out jobs the moment the deadline passes, and check aborts its in-flight HTTP request because the request carries the same context.

        The "first answer wins" tab shows a smaller detail that is easy to get wrong. The results channel has room for every mirror's answer, so the losing goroutines can always complete their send and exit, even though nobody reads their result. With an unbuffered channel they would block forever. That is the first bug in the next level.

        Check yourselfIn the worker pool above, which goroutine should close the results channel?

        Level 04/08Interface

        Bugs they invite

        Go makes concurrency cheap to write, which also makes concurrency bugs cheap to write. The compiler checks channel element types and directions, and almost nothing else: it does not know whether a goroutine will ever stop, whether two goroutines touch the same variable, or who is supposed to close a channel. Here are the bugs you will actually meet, roughly in order of how often.

        Goroutine leaks

        A goroutine that blocks forever is never collected. The garbage collector does not reclaim goroutines, only memory nothing can reach, and a blocked goroutine's stack is a root: everything it references stays alive too. A leak of one goroutine per request is invisible in tests and fatal after a week in production.

        leaks len(mirrors) - 1 goroutines per call
        func fastest(mirrors []string) string {
            ch := make(chan string) // unbuffered
            for _, m := range mirrors {
                go func() {
                    check(m)
                    ch <- m // only the winner is ever received
                }()
            }
            return <-ch
        } // every other goroutine blocks on its send forever
        give every send somewhere to go
        ch := make(chan string, len(mirrors)) // every send has a slot
        
        // or, when the work is open-ended, give every send an exit:
        select {
        case ch <- m:
        case <-ctx.Done():
        }

        To find leaks, watch runtime.NumGoroutine() over time, read the goroutine profile from net/http/pprof (thousands of goroutines parked on the same line is the signature), and use a checker such as go.uber.org/goleak in tests. Recent releases have also been experimenting with a runtime goroutine leak profile that uses the garbage collector to find goroutines blocked on channels no one else can reach.

        Deadlocks

        If every goroutine in the program is blocked on a channel or lock and nothing else could wake them (no timers, no I/O in flight), the runtime notices and kills the process with a message every Go programmer meets in their first week:

        main.go
        func main() {
            ch := make(chan int)
            ch <- 1 // no receiver can ever exist
            fmt.Println(<-ch)
        }
        output
        fatal error: all goroutines are asleep - deadlock!
        
        goroutine 1 [chan send]:
        main.main()
                /tmp/x/main.go:7 +0x2d
        exit status 2

        Data races

        A data race is two goroutines accessing the same memory at the same time, at least one of them writing, with nothing ordering them (no channel operation, lock or atomic in between). The canonical example is a shared counter. hits++ looks like one step, but it is three: load the value, add one, store it back. On amd64 the compiler may even emit it as a single INCQ instruction, and it is still not atomic without a LOCK prefix. Choose the interleaving yourself:

        G1 runs counter++

        1. r = counter
        2. r = r + 1
        3. counter = r

        register r ?

        G2 runs counter++

        1. r = counter
        2. r = r + 1
        3. counter = r

        register r ?

        shared memory

        counter0

        expected after both: 2

        Two goroutines increment the same counter. Run their steps in any order. Any order where both loads happen before either store loses an update.

        The race detector finds these for you. Build or test with -race and the compiler instruments every memory access; a runtime library based on ThreadSanitizer tracks which goroutine touched each word and what synchronisation happened in between, and reports conflicting accesses with both stack traces:

        racy.go
        var hits int
        var wg sync.WaitGroup
        for range 1000 {
            wg.Go(func() { hits++ })
        }
        wg.Wait()
        fmt.Println(hits) // 1000? Usually. Sometimes 987.
        go run -race
        $ go run -race main.go
        ==================
        WARNING: DATA RACE
        Read at 0x00c000012118 by goroutine 8:
          main.main.func1()
              /tmp/x/main.go:10 +0x2c
        
        Previous write at 0x00c000012118 by goroutine 7:
          main.main.func1()
              /tmp/x/main.go:10 +0x3e
        ==================

        It only reports races that actually happen during the run, so it is only as good as your tests. It costs roughly 5 to 10 times the memory and 2 to 20 times the CPU, so run it in CI and in a canary, not everywhere. And take its reports seriously: Go's memory model limits the damage of a race on a single machine word, but a race on a multi-word value (a string, slice, interface or map) can tear it in half and corrupt memory. Maps even have a cheap built-in check that crashes with fatal error: concurrent map writes.

        The loop variable that every closure shared

        the Go 1.22 change
        var wg sync.WaitGroup
        for _, u := range []string{"a", "b", "c"} {
            wg.Add(1)
            go func() {
                defer wg.Done()
                fmt.Print(u, " ")
            }()
        }
        wg.Wait()
        // go 1.21 and earlier: often prints "c c c"
        // one variable u, shared by every iteration and every closure
        
        // go 1.22 and later (go.mod says go 1.22+): prints a, b, c in some order
        // each iteration gets its own u

        Until Go 1.21, a for loop declared its variables once and reused them for every iteration, so every closure that captured u captured the same variable. By the time the goroutines ran, the loop had usually finished and u held the last value. This was probably the most common Go concurrency bug for a decade. When the Go team tried the new semantics on Google's own code base, almost every failure it turned up was a test that had been silently passing while testing nothing. Go 1.22 changed the semantics so that each iteration gets a fresh variable. The change applies per module, based on the go line in go.mod, so old modules keep the old behaviour until they opt in.

        Closing twice, or closing under a sender

        two owners, one channel
        // two producers, each "helpfully" closing when done
        go func() { produce(ch, part1); close(ch) }()
        go func() { produce(ch, part2); close(ch) }()
        // panic: close of closed channel
        // or, if one closes early: panic: send on closed channel
        one owner closes, after all senders finish
        var wg sync.WaitGroup
        wg.Go(func() { produce(ch, part1) })
        wg.Go(func() { produce(ch, part2) })
        go func() {
            wg.Wait()
            close(ch) // one owner closes, after every sender is done
        }()

        There is no way to ask "is this channel closed?" without receiving from it, and good reason: the answer could change the instant after you got it. The fix is a convention, not an API. A channel has exactly one owner, the owner is the only one who closes it, and receivers never close. With several senders, the owner is whoever waits for all of them.

        Check yourselfAn HTTP server has two handler goroutines that one locking mutex A then B, the other B then A, and they deadlock. What does the Go runtime do?

        Level 05/08Runtime

        The G-M-P scheduler

        So far a goroutine has been a black box that "runs concurrently". Time to open it. The link checker has 1,000 goroutines and a machine with, say, 8 cores. Something has to decide which goroutine runs on which core, and switch between them when one blocks. In a thread-based program that something is the kernel. In Go it is the runtime's scheduler, in runtime/proc.go, and it lives entirely in user space.

        Go 1.0 had a simple scheduler with one global run queue guarded by one lock. It worked, but every scheduling decision on every core fought over that lock. Dmitry Vyukov's redesign, which shipped in Go 1.1, introduced the three-letter model the runtime still uses.

        G, M and P

        • G is a goroutine: a runtime.g struct with its stack bounds, its saved registers (stack pointer, program counter and a few more, in g.sched) and its status (runnable, running, waiting, in a syscall, dead).
        • M is a machine: an OS thread. It is what actually executes instructions.
        • P is a processor: a permit to run Go code, plus the resources that go with it. Each P has a local run queue of goroutines, a cache for the memory allocator, and other per-P state. An M must hold a P to run Go code. There are exactly GOMAXPROCS Ps.

        Why the middle letter? Because threads block. When a goroutine makes a blocking system call, the kernel freezes its thread, and nothing can be done about that. Without P, the run queue and caches attached to that thread would be frozen with it. With P, the runtime can detach the P from the stuck M and hand it to another thread, so the queued goroutines keep running. The number of threads can exceed GOMAXPROCS, but no more than GOMAXPROCS of them run Go code at once.

        GOMAXPROCS defaults to the number of CPUs the process may use. Since Go 1.25, on Linux, that includes the container's cgroup CPU limit, and the runtime re-checks it periodically, so a Go service in a container limited to 2 CPUs no longer starts 64 Ps on a 64-core host.

        What go compiles to

        runtime.newproc, heavily simplified
        // what "go check(u)" turns into, roughly
        newproc(fn) // runtime/proc.go
            newg := gfget(pp)       // reuse a dead G from this P's free list...
            if newg == nil {
                newg = malg(stackMin) // ...or allocate one with a fresh small stack
            }
            // fake a call frame so that when fn returns, it "returns" into goexit
            newg.sched.pc = abi.FuncPCABI0(goexit) + sys.PCQuantum
            gostartcallfn(&newg.sched, fn)
            casgstatus(newg, _Gdead, _Grunnable)
            runqput(pp, newg, true) // true: into the runnext slot
            if mainStarted {
                wakep() // an idle P? get a thread spinning on it
            }

        No system call, no kernel involvement: grab a g (recycled if possible), point its saved PC at the function, put it in the runnext slot of the current P. The slot means "run this one next": a goroutine that just created or woke another often wants to hand it data, so running it soon keeps the data hot in cache. Whatever was in runnext moves to the tail of the local queue.

        Finding the next goroutine

        Whenever a goroutine blocks, yields, or finishes, its M calls schedule(), which calls findRunnable(). In order, simplified:

        1. Every 61st scheduling round, take one goroutine from the global queue first, so it can't be starved by busy local queues.
        2. Take runnext, then the head of the P's local queue: a 256-slot ring buffer that only its owner adds to, so it needs no lock.
        3. Take a batch from the global queue (a fair share, divided by the number of Ps), which needs the scheduler lock.
        4. Ask the network poller whether any goroutines are now ready (the threads level explains this).
        5. Steal: pick other Ps in random order and take half of the first non-empty local queue found, using atomic operations. This is how a P that runs dry rebalances itself with no central coordinator.
        6. Nothing anywhere? Put the P on the idle list and park the thread.

        Overflow goes the other way: if a P's local queue is full when a goroutine is added, half of it moves to the global queue in one batch. And a thread that is about to park first spins for a while, looking for work, because parking and waking a thread costs microseconds. The runtime caps how many threads may spin at once so idle machines don't burn CPU.

        Now drive it yourself. Each step is 2 ms of simulated time. Every goroutine has a random amount of work.

        Global run queue

        Threads without a P

        Scheduler log

          A G-M-P scheduler simulation, simplified but following the rules in runtime/proc.go. Things to try: spawn 12 with GOMAXPROCS 3 and step, to watch idle Ps steal; block a goroutine in a syscall while P0 has a queue, to watch sysmon hand its P to a new thread; spawn 300 to overflow P0's 256-slot queue into the global queue; let a long goroutine run five steps to see it preempted.

          sysmon, the watchdog

          One runtime thread runs without a P at all: sysmon. It sleeps 20 µs at first, backing off to 10 ms when there is nothing to do, and on each wake-up it does housekeeping that nobody else is in a position to do:

          • Retake Ps from syscalls. If a P has been sitting with a thread stuck in a syscall for a sysmon tick and there is other work it could do, sysmon hands it to another M (waking a parked thread or creating a new one). Short syscalls usually return before this happens and take the fast path: the thread just keeps its P.
          • Preempt hogs. If a goroutine has been running for more than 10 ms, sysmon asks it to yield. The last level shows the two mechanisms behind "asks".
          • Poll the network if nobody has for 10 ms, so I/O-ready goroutines aren't left waiting behind CPU-bound ones.

          When a syscall returns and its P has been taken, the goroutine tries to grab any idle P. If none is free, it goes to the global queue and the thread parks, because a thread without a P is not allowed to run Go code.

          watching the real scheduler (exact fields vary by release)
          $ GODEBUG=schedtrace=1000 ./checklinks
          SCHED 0ms: gomaxprocs=4 idleprocs=2 threads=5 spinningthreads=1 needspinning=0 idlethreads=0 runqueue=0 [0 0 0 0]
          SCHED 1004ms: gomaxprocs=4 idleprocs=0 threads=9 spinningthreads=0 needspinning=0 idlethreads=3 runqueue=17 [44 3 0 12]

          GODEBUG=schedtrace=1000 prints a line like this every second: the global runqueue length, then each P's local queue length in brackets. Add scheddetail=1 for every G, M and P. For a timeline, go tool trace on a runtime/trace capture shows each P as a lane and every goroutine switch on it.

          Check yourselfWith GOMAXPROCS=4, 50 goroutines are all blocked in slow file reads (blocking syscalls). Roughly how many OS threads does the process have?

          Level 06/08Runtime

          Inside a channel

          A channel operation looked like magic at the interface level: values appear in other goroutines, blocked goroutines wake up at the right moment. With the scheduler in hand, it turns out to be a short, readable piece of code in runtime/chan.go. A channel is a struct on the heap, protected by a lock, holding a ring buffer and two queues of parked goroutines.

          runtime/chan.go
          // runtime/chan.go (recent releases, abridged; fields shift between versions)
          type hchan struct {
              qcount   uint           // values currently in buf
              dataqsiz uint           // capacity: make(chan T, N) → N
              buf      unsafe.Pointer // ring of dataqsiz elements
              elemsize uint16
              closed   uint32
              timer    *timer         // set for time.Timer / Ticker channels
              elemtype *_type
              sendx    uint           // where the next send writes
              recvx    uint           // where the next receive reads
              recvq    waitq          // parked receivers (sudogs)
              sendq    waitq          // parked senders (sudogs)
              bubble   *synctestBubble // testing/synctest support
          
              lock mutex // guards every field above
          }
          
          type sudog struct {
              g    *g             // the parked goroutine
              next *sudog
              prev *sudog
              elem unsafe.Pointer // the value to send, or where to put the received one
              // ... plus fields for select, timing and the channel it waits on
          }

          A sudog represents a goroutine waiting on a channel. It exists because one goroutine can wait on many channels at once (in a select) and many goroutines can wait on one channel, so the wait can't simply be a field on g. The important field is elem: a pointer to the value being sent, or to the variable that will receive, and it usually points into the waiting goroutine's own stack.

          Step through the life of one buffered channel and watch the fields change:

          hchan on the heap

          buf circular, 3 slots

          sendq parked senders

          recvq parked receivers

          1. make(chan int, 3) allocates one hchan on the heap plus a 3-element array for buf. The variable ch is just a pointer to it, which is why passing a channel around never copies the queue.
          2. G1 sends 10. It takes the lock, sees no parked receiver and free room (qcount < dataqsiz), copies 10 into buf[sendx], bumps sendx and qcount, and unlocks. No scheduler involved.
          3. Two more sends fill the buffer. sendx wraps from 2 back to 0: the buffer is a ring, so the next write position is the slot the next read will free.
          4. G1 sends 40 into a full buffer. It wraps itself in a sudog (a "goroutine waiting on something" record) whose elem points at the 40 on G1's own stack, appends it to sendq, and calls gopark. The lock is released as part of parking, and G1's P runs something else.
          5. G2 receives. The buffer is full and a sender is waiting, so recv does two copies in one critical section: the head value 10 goes to G2, then G1's 40 drops into the slot just freed. recvx and sendx both advance to 1, and goready(G1) puts G1 back in a run queue. FIFO order is preserved.
          6. G2 keeps receiving: 20, then 30, then 40. Each is a plain locked copy out of buf[recvx]. Now qcount is 0 and the indices sit together at 1.
          7. G3 receives from the empty channel. It parks in recvq, its sudog.elem pointing at the variable on its own stack where the value should land.
          8. G4 sends 50 and finds G3 waiting. It skips the buffer and copies 50 straight onto G3's stack through sudog.elem, then readies G3. The runtime source calls this out as the only place one goroutine writes to another goroutine's stack.
          9. close(ch) sets closed under the lock and wakes every parked goroutine: receivers get the zero value with ok == false, senders panic. Later receives drain anything left in buf, then return zero values forever.
          1 / 9
          runtime.hchan for make(chan int, 3), step by step. Highlighted fields are the ones the current operation touches. sendx and recvx are indices into the ring.

          Three paths for a send

          chansend takes the lock and checks, in this order:

          1. Is a receiver already parked? Copy the value straight to it and make it runnable, skipping the buffer even on a buffered channel. On an unbuffered channel this is how a send completes when the receiver arrived first. If the sender arrives first it takes path 3 and parks, and the receiver later copies the value out of the sender's sudog. Either way, an unbuffered value never touches a buffer.
          2. Is there room in the buffer? Copy into buf[sendx], advance sendx, unlock. The common fast path for buffered channels: no scheduler involvement at all.
          3. Otherwise, park. Wrap the goroutine in a sudog, append it to sendq, and call gopark, which releases the channel lock only after the goroutine is safely off the CPU, so no wake-up can be lost in between.

          Receives mirror this. A send on a closed channel panics before any of it, and a send on a nil channel parks forever without even taking a lock.

          How select works

          selectgo, in runtime/select.go, shuffles the cases into a random poll order (that is where the fairness comes from), then locks every channel involved, sorted by address, so two selects over the same channels in different orders can't deadlock each other. It makes one pass looking for a case that can proceed right now. If none can, it enqueues a sudog on every channel, parks, and when it is woken by one of them, dequeues itself from all the others.

          A channel is a mutex, a ring buffer and two wait queues. Blocking on it is just the scheduler parking a G, which is why it is cheap.

          Check yourselfOn an unbuffered channel, G1 is parked in recvq. G2 sends 42. Where is 42 written?

          Level 07/08System

          OS threads, syscalls and the netpoller

          One level down, the Ms are ordinary OS threads. On Linux the runtime creates them with the clone system call (in runtime/os_linux.go), sharing the address space, file descriptors and signal handlers like any pthread. Each M also has two goroutines of its own that you never see: g0, whose stack the scheduler itself runs on, and gsignal, for signal handlers.

          Which raises the question this level is about. The link checker has 1,000 goroutines waiting on sockets at once. If each of those waits were a blocking read system call, each would pin an M in the kernel, and the program would need 1,000 threads, exactly the cost goroutines were meant to avoid. In practice it uses a handful. The difference is the network poller.

          1. Your goroutineHandler goroutine G7 callsn, err := conn.Read(buf)
          2. Go runtimeinternal/poll.FD.Read issues the raw syscall on a non-blocking socketread(fd=9, …)
          3. KernelNo bytes yet. The kernel returns at once instead of sleeping the thread-1 EAGAIN
          4. Go runtimepollDesc.waitRead parks G7 and records it in the descriptor's read slot. M0 is freegopark · pd.rg = G7
          5. Go runtimeM0 runs other goroutines from its P. Thousands of connections can wait this way on a handful of threads
          6. KernelA packet arrives. The fd, registered once when the socket was created, lands on epoll's ready listEPOLLIN | EPOLLET
          7. Go runtimeA scheduler with no local work (or sysmon, if none has polled for 10 ms) asks the kernel what is readynetpoll(0) → epoll_wait
          8. Go runtimeThe ready fd maps back to its pollDesc, and from there to G7. G7 goes into a run queuenetpollready · injectglist
          9. Your goroutineG7 resumes inside Read, retries the syscall and gets the bytes. To the code it was one blocking calln = 512, err = nil
          1. A goroutine reads from a TCP connection. The code looks blocking, and from the goroutine's point of view it is. Underneath, every socket the net package creates is set to non-blocking mode.
          2. The kernel has nothing buffered for this socket, so a non-blocking read fails immediately with EAGAIN instead of putting the thread to sleep.
          3. The runtime parks only the goroutine. Its pollDesc (one per file descriptor) remembers which goroutine is waiting to read. The OS thread is not blocked.
          4. The thread goes back to the scheduler loop and runs other goroutines. This is the whole trick: waiting costs a parked g and a few hundred bytes of bookkeeping, not a thread.
          5. Data arrives. Because the fd was added to the epoll instance once, edge-triggered, the kernel just marks it ready. No goroutine or thread is spending CPU on it.
          6. The runtime polls without blocking when a P runs out of work in findRunnable, and sysmon polls too if nobody has for 10 ms. One idle thread may also block in epoll_wait when there is nothing else to do.
          7. Each event carries a pointer back to its pollDesc. The waiting goroutine is pulled out of the read slot and made runnable.
          8. G7 wakes up inside Read, loops back to the syscall, and this time the kernel has data. The same flow uses kqueue on BSD and macOS and I/O completion ports on Windows.
          1 / 8
          How conn.Read waits without blocking a thread. Three lanes: the goroutine's view, the runtime, and the kernel.

          The netpoller wraps whatever the OS offers for "tell me which of these many descriptors is ready": epoll on Linux (runtime/netpoll_epoll.go), kqueue on the BSDs and macOS, I/O completion ports on Windows, event ports on Solaris and illumos. A socket is registered once, when it is created, and stays registered until it is closed. Waiting becomes a matter of parking a goroutine on its pollDesc; readiness becomes a matter of making that goroutine runnable again. Nothing in between costs a thread.

          This is how Go solves the Dan Kegel's 1999 name for the problem of serving 10,000 concurrent connections on one machine, which thread-per-connection servers of the time could not. while letting net/http use one goroutine per connection with plain blocking code. The event loop that a Node.js or nginx programmer writes by hand is still there. The runtime writes it for you.

          What blocks what

          Not everything can go through the poller. Here is what each kind of wait actually costs:

          The goroutine waits onGoroutineThread (M)P
          Channel, select, sync.Mutex, WaitGroupparkedfree, runs other Gsstays
          time.Sleep, timersparked on the P's timer heapfreestays
          Sockets, pipes (network I/O)parked on the netpollerfreestays
          Regular file I/O on Linuxin a syscallblocked in the kernelhanded off if it takes long
          cgo call into Cin C codeblocked in Chanded off if it takes long
          After runtime.LockOSThreadruns only on its Mdedicated to itnormal

          Regular files are the odd one out because epoll does not support them: the kernel considers a disk file always ready, so there is nothing to wait for, and epoll_ctl refuses to register one. The os package tries the poller, and falls back to plain blocking syscalls with P handoff. That is why a program that reads thousands of files concurrently can create thousands of threads, while one with thousands of idle connections stays at a handful.

          runtime.LockOSThread exists for the cases where the OS cares which thread you are on: GUI toolkits that must run on the main thread, thread-local state in C libraries, Linux namespaces set with setns. It wires the goroutine to its M until it unlocks.

          Check yourselfA Go WebSocket server holds 50,000 idle connections, each with a goroutine blocked in conn.Read, on an 8-core machine. About how many OS threads does it use?

          Level 08/08Machine

          Growable stacks and preemption signals

          One question has been left open since the comparison table. How can a goroutine start with a 2 KB stack when C threads need megabytes? A deep recursion, a large local array or a chain of library calls would run off the end of 2 KB in no time. The answer is the bottom of the stack: two instructions at the start of nearly every Go function, and a runtime willing to move a running goroutine's entire stack somewhere else.

          Every function checks before it grows the stack

          The compiler adds a The instructions at the start of a function that set up its stack frame, before the body runs. like this to almost every function (shown for amd64, simplified slightly):

          go build -gcflags=-S, amd64, Go 1.17+ register ABI (simplified)
          TEXT main.walk(SB), ABIInternal, $40-8
              CMPQ  SP, 16(R14)    // R14 = current g; g+16 = g.stackguard0
              JLS   grow           // SP <= guard (unsigned): not enough room
              SUBQ  $40, SP        // normal path: allocate the frame
              ...                  // function body
              ADDQ  $40, SP
              RET
          grow:
              MOVQ  AX, 8(SP)      // spill the register argument
              CALL  runtime.morestack_noctxt(SB)
              MOVQ  8(SP), AX
              JMP   main.walk(SB)  // start over on the new, bigger stack

          Since Go 1.17's register-based calling convention, amd64 code keeps the current goroutine's g pointer in register R14. The first fields of runtime.g are the stack bounds (stack.lo at offset 0, stack.hi at 8) followed by stackguard0 at offset 16: a limit set a little above the bottom of the stack. Stacks grow down, so if the stack pointer is at or below the guard, this function's frame might not fit, and it jumps to morestack. Here is the comparison as the CPU sees it:

          49010010013B0011101166011001101000010000
          1. REX prefix0100 W=1 (64-bit) R=0 X=0 B=1 (base register is r8-r15)
          2. OpcodeCMP r64, r/m64
          3. ModRMmod=01 (8-bit displacement), reg=100 (rsp), rm=110 (+B = r14)
          4. Displacement+16: offsetof(g, stackguard0)
          CMPQ SP, 16(R14) is the Intel instruction cmp rsp, [r14+0x10], encoded in four bytes. JLS ('jump if lower or same', an unsigned comparison) becomes jbe.

          A compare and a predictable, almost-never-taken branch: about the cheapest check there is. Small leaf functions (no calls, tiny frames) skip it, because they can safely fit in a reserved zone below the guard. When the check fails, morestack switches to the scheduler stack, allocates a new stack twice the size, copies the old one over, and restarts the function, which now passes the check.

          Copying a running stack

          In use
          Stack size
          Times grown
          Bytes copied in total
            A goroutine recursing ever deeper; the stack fills downward from its high end. Drag the slider; stack sizes and copy costs are computed by doubling from 2 KB whenever the frames no longer fit (simplified: the real check leaves a guard zone, and the starting size can be larger, see below).

            Copying sounds expensive, but doubling makes it cheap overall: the total bytes ever copied stay below the final stack size, the same amortised argument as for growing a slice. The hard part is not the copy, it is the pointers. Any pointer that pointed into the old stack (a local variable holding the address of another local, a saved frame pointer) must be adjusted to point into the new one.

            Two guarantees make that possible. First, the compiler emits stack maps for every function: for each point where the stack might be copied, which slots in the frame hold pointers. The garbage collector needs those anyway. Second, escape analysis guarantees that nothing on the heap points into a stack: if the address of a local could outlive the frame or be shared, the variable is allocated on the heap instead. So the runtime only has to scan the stack itself, adjusting pointers that fall in the old range. The exceptions are runtime-internal, and you have already met the main one: a sudog.elem pointing into a parked goroutine's stack. That is why the channel code is so careful about when another goroutine may write through it.

            C can't do this. It has no stack maps, and a pointer to a stack variable can be stored anywhere, so a C thread's stack must stay where it is, and must be big enough up front.

            A short history

            • Up to Go 1.2, stacks were segmented: when one ran out, a new segment was linked on, and freed when the function returned. A loop calling a function right at a segment boundary would allocate and free a segment on every iteration, the "hot split" problem, which made performance erratic.
            • Go 1.3 switched to contiguous stacks, grown by copying, as described above.
            • Go 1.4 dropped the starting size from 8 KB to 2 KB, now that growing was cheap and predictable.
            • Go 1.19 made the starting size adaptive: the runtime tracks average stack usage at each garbage collection and starts new goroutines with that size, so typical goroutines skip the first few rounds of growth. 2 KB is the minimum.

            Stacks also shrink: during garbage collection, a goroutine using less than a quarter of its stack gets it halved. The upper bound is 1 GB on 64-bit systems, after which the program dies with goroutine stack exceeds 1000000000-byte limit, which is how infinite recursion shows up in Go.

            The same check, reused for preemption

            Now the last piece of the scheduler falls into place. When sysmon wants a goroutine to yield, it needs it to stop at a point where the GC knows exactly which registers and stack slots hold pointers. Function prologues are such points, and they already have a check. So since Go 1.2 the runtime "asks" a goroutine to yield by setting its stackguard0 to a poison value, stackPreempt, that is larger than any real stack address. The next prologue's comparison fails, morestack runs, notices the poison, and calls the scheduler instead of growing anything.

            That covers any goroutine that calls functions. A goroutine in a tight loop with no calls never runs a prologue, and before Go 1.14 it could stall the garbage collector, and with it the whole program. Go 1.14 added a second mechanism that interrupts the thread itself:

            1. M2 running G9G9 spins in a loop with no function calls, so no prologue check ever runsfor { sum += x[i&mask]; i++ }
            2. sysmon (no P)sysmon notices P2 has run the same goroutine for more than 10 msretake → preemptone(pp)
            3. sysmon (no P)The polite request: set the preempt flag and poison the stack guardgp.stackguard0 = stackPreempt
            4. sysmon (no P)Since Go 1.14 it does not wait to see if that works: it signals the thread tootgkill(tid, SIGURG)
            5. Kernel and signal handlerThe kernel interrupts M2 mid-loop and runs Go's signal handler on the signal stacksighandler → doSigPreempt
            6. Kernel and signal handlerIf the interrupted PC is an async safe point, the handler rewrites the saved context to fake a callctxt.pushCall(asyncPreempt)
            7. M2 running G9The signal returns into asyncPreempt, which spills every register to G9's stack and yieldsasyncPreempt → gopreempt_m
            8. M2 running G9Later G9 is rescheduled. Registers are restored and the loop continues exactly where it wasRET to the interrupted PC
            1. A goroutine is stuck in a tight loop. Before Go 1.14 this could hang a whole program: the garbage collector's stop-the-world phase would wait forever for this goroutine to reach a safe point.
            2. sysmon is a runtime thread that runs without a P. It wakes every 20 µs to 10 ms and checks each P's schedtick; if it has not changed for over 10 ms, the current goroutine has been hogging the P.
            3. First it asks nicely. Setting stackguard0 to stackPreempt, a value larger than any real stack address, makes the next function prologue's check fail. morestack then sees the request and yields instead of growing the stack. This piggybacking on the stack check is how cooperative preemption has worked since Go 1.2.
            4. A loop without calls has no prologue to trip over, so in the same breath preemptone sends SIGURG to that specific thread. It picked SIGURG because programs almost never use it for anything and its default action is to be ignored.
            5. The kernel saves M2's registers into a signal context and enters the runtime's handler, which recognises the preemption signal.
            6. Not every instruction is safe to stop at (the GC needs to know which registers and stack slots hold pointers). At a safe point the handler edits the saved context: push the interrupted PC, set PC to asyncPreempt. If the PC is not safe, it just returns and sysmon tries again later.
            7. When the kernel resumes the thread it lands in asyncPreempt, an assembly routine that saves all registers, then calls into the scheduler. G9 goes to the global run queue and the P runs someone else.
            8. When G9 is picked again, asyncPreempt restores the registers and returns to the saved PC. The loop never knew. You can switch this off with GODEBUG=asyncpreemptoff=1, mostly for debugging.
            1 / 8
            Asynchronous preemption with SIGURG, as implemented in runtime/preempt.go and the signal handler in runtime/signal_unix.go.

            That is the bottom of the stack for Go concurrency. A go statement allocates (or reuses) a g with a small stack and puts it in a P's runnext slot. A channel operation takes a runtime lock and either copies a value or parks a g on a wait queue. Network waits park on the netpoller instead of a thread. Ms are ordinary OS threads that pick up whatever the Ps hand them. And underneath all of it, a four-byte cmp at the top of nearly every function lets stacks start small, grow on demand, and doubles as the hook that keeps any one goroutine from hogging a core.

            Goroutines are cheap because the compiler and runtime cooperate: the compiler emits a stack check and stack maps, so the runtime can start stacks tiny, move them, and interrupt code safely.

            Check yourselfWhy can the Go runtime move a goroutine's stack to a new address, when a C runtime cannot move a thread's stack?

            End of Go concurrency · 8 levels, Idea to Machine

            Where to next

            Practical notes

            1. Note3 min read

              Find and fix a goroutine leak in Go

              Compare goroutine stacks after work has drained, identify the missing exit path, and cancel and join workers before returning.

            Browse all 34 divesAll topics