Infrastructure and operationsDive 01 of 01

From kubectl apply to a ready Pod

Trace one Deployment through the API server, controllers, scheduler and kubelet. Follow a real Pod into a Service and diagnose three places startup can stall.

Levels
8
Reading
22 min
Depth
Idea to System
Published
  1. API serverDeployment recorded
  2. ControllersReplicaSet and Pod created
  3. SchedulerPod bound to a node
  4. KubeletContainer started and probed
  5. EndpointSlice controllerService backend marked ready
Each handoff leaves a different record in the Kubernetes API.

Level 01/08Idea

A Deployment describes what should exist

You run kubectl apply and see deployment.apps/web created. The Pod eventually reports Running while the Service still has zero ready backends. Those observations refer to different state changes, made by different components. Which one is your request waiting for?

Follow one Deployment named web. Its selector identifies the workload and its Pod template describes what to instantiate. The replica count is one. The Service is a separate object with its own selector, app: web. It maps port 80 to the named container port http. The HTTP readiness probe asks for /ready.html, which the stock NGINX image lacks. The lab creates that file after observing the first failure. Deployments · Services

The Deployment and Service manifest
workload.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /ready.html
              port: http
            periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: http
Observed stdout · kubectl apply -f workload.yaml
deployment.apps/web created
service/web created

The Deployment declares a desired set of Pods. The Service independently selects Pods that can serve its traffic.

Level 02/08Interface

Apply writes to the API

In this run, kubectl used client-side apply. It decoded the manifest, found the API resource for each object and looked up its live state. With no Deployment of that name it sent a create request and stored the configuration in the kubectl.kubernetes.io/last-applied-configuration annotation. On later changes, it calculates a patch using the manifest, the live object and that previous configuration. With --server-side, the server handles the apply merge and records field ownership in managedFields. Client-side merge · Server-Side Apply

The API server authenticates the caller, authorizes the write, and runs admission and API validation before persistence. Defaults can fill unset fields, and admission can change or reject the request. The Deployment and Service are separate writes. A later failure applying the Service leaves the first object in place. Kubernetes commonly uses etcd for API storage. This K3s cluster used its default SQLite datastore behind Kine, which provides an etcd-compatible API. Admission controllers · K3s datastore options · Kine

Deployment create · fields from the API server audit log
POST /apis/apps/v1/namespaces/pod-dive/deployments
HTTP 201
metadata.uid = 3fe3a7e6-9c45-4a9b-bec8-44f87f714c13

The create response acknowledges the stored Deployment. Workload controllers can act on it independently of the client.

Level 03/08Runtime

Two controllers turn a template into a Pod

Controllers use the API to observe changes and work toward the state described by each object. The Deployment controller finds a ReplicaSet for the current Pod template or creates one. It adds the pod-template-hash label to the ReplicaSet selector and its Pod template, so ReplicaSets for different revisions select distinct Pods. Changing the Pod template triggers a rollout. Changing only the replica count scales the applicable ReplicaSets without producing another template revision. Deployment rollouts

With a desired count of one and zero matching controlled Pods, the ReplicaSet controller creates a Pod from its template. Each object records its controlling owner through metadata.ownerReferences. In the trace you can follow the UID in the Pod's owner reference to the ReplicaSet, then follow the ReplicaSet's owner UID to the Deployment. A label is a selection rule. The owner UID identifies a particular object even if another object later uses the same name. ReplicaSet reconciliation · Owners and dependents

Observed owner references · tree assembled from create responses
web
  Deployment UID       3fe3a7e6-9c45-4a9b-bec8-44f87f714c13
└─ web-6d5c46b4dc
   ReplicaSet UID      9a867ace-fc55-4ab4-a939-322490b7dcdb
   owner UID           3fe3a7e6-9c45-4a9b-bec8-44f87f714c13
   └─ web-6d5c46b4dc-5qkld
      Pod UID          c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904
      owner UID        9a867ace-fc55-4ab4-a939-322490b7dcdb

The controller watches API objects and queues keys for reconciliation. A watch notification tells the controller what to revisit. Its next pass evaluates the current state it can observe, so repeated notifications can lead to a single pass. If this Pod disappears and the ReplicaSet still wants one replica, it creates a replacement with a different UID. A container restart inside a surviving Pod keeps that Pod's identity. Controller loops · ReplicaSet controller source · Pod lifetime

Deployment → ReplicaSet → Pod is a chain of independently reconciled API objects, with owner references you can inspect.

Level 04/08Runtime

The scheduler binds the Pod to a node

The fresh Pod has no spec.nodeName. The scheduler pulls an eligible, unassigned Pod from its queue and filters out nodes that fail its constraints. Depending on the Pod, those include resource requests, selectors, required affinity, taints and storage topology. The scheduler scores feasible nodes when there is a choice. CPU fit uses requested resources against allocatable capacity and already assigned requests. Low CPU usage in a metrics dashboard does not make a request fit. Scheduling and binding · Resource requests and scheduling

In our single-node cluster, pod-dive-node passed the filters. Through the Pod's binding subresource, the scheduler submitted a target node and the Pod UID. The API server recorded the assignment as spec.nodeName. That stored field is the handoff to the node agent. If the scheduler cannot place the Pod, the assignment remains empty and you can inspect PodScheduled and the scheduler's Events. Default binder source

Observed binding · selected request fields
POST /api/v1/namespaces/pod-dive/pods/web-6d5c46b4dc-5qkld/binding
HTTP 201
metadata.uid = c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904
target.name  = pod-dive-node

Scheduling picks a node. The stored binding assigns the Pod to that node so its kubelet can act.

Level 05/08System

The kubelet starts the containers

The kubelet sees the Pod assigned to its node and works to make the actual containers match the spec. It handles prerequisites such as required volumes, then asks the container runtime through the The gRPC API through which the kubelet manages Pod sandboxes, containers and images. to create a Pod sandbox. In this setup, containerd runs the sandbox and invokes the CNI networking plugin. The sandbox supplies the shared network namespace and Pod IP used by containers in that Pod. In Kubernetes 1.34, PodReadyToStartContainers reports sandbox and network readiness and is enabled by default. Container Runtime Interface · Pod network readiness

The kubelet ensures the image is available under its pull policy, then requests container creation and start. This Pod has no init containers and our NGINX image was preloaded. The Events show it is already present, then that the container was created and started. The Pod status patch reports Running, IP 10.42.0.2 and Ready: False. An HTTP 404 is enough to show a process answering on the probe port. It also shows why the readiness probe has yet to pass. Image pull policy · Probe results

Observed Events · kubelet
Pulled     Container image "nginx:1.27-alpine" already present on machine
Created    Created container: nginx
Started    Started container nginx
Unhealthy  Readiness probe failed: HTTP probe failed with statuscode: 404

Running is a coarse Pod phase. Per-container Waiting, Running and Terminated states tell you what particular processes are doing. The STATUS column in kubectl get pods can show display reasons such as ContainerCreating or ImagePullBackOff. To distinguish them from the phase and readiness conditions, inspect the object. Pod phases and container states

A node assignment, a prepared sandbox, a running container and a ready Pod represent different milestones.

Level 06/08System

Readiness changes the Service’s backends

The kubelet's HTTP probe goes to the Pod IP on the configured port. It bypasses the Service's virtual IP. In the lab, writing /usr/share/nginx/html/ready.html makes that probe pass. A failed readiness probe changes readiness without restarting the process. A liveness failure can trigger a restart. A startup probe, when configured, holds off readiness and liveness checks until startup succeeds. At the Pod level, any configured readiness gates also need to be true. Readiness, liveness and startup probes · Pod readiness gates

The EndpointSlice controller watches matching Pods and publishes endpoint addresses and conditions for the Service. Our slice already held the Pod's address with ready: false. After the kubelet reported Ready=True, the controller updated that same endpoint to ready: true. Consumers such as kube-proxy watch the slices and update their forwarding rules. In this lab, an HTTP request through the Service ClusterIP returned the file after the endpoint became ready. The API write and local forwarding update are separate steps. The Service option publishNotReadyAddresses can deliberately publish endpoints as ready regardless of Pod readiness. EndpointSlice conditions · Service virtual IPs

Observed EndpointSlice snapshots and HTTP response
same EndpointSlice  web-qsrw8
same endpoint IP    10.42.0.2
same target UID     c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904
conditions.ready    false → true

GET http://10.43.178.85/ready.html
ready

Deployment availability is reported separately. A newly ready Pod can already be a Service backend while the Deployment still waits out its minReadySeconds before counting it as available. That value defaults to zero in our manifest. Deployment availability timing

Pod readiness is kubelet status. An EndpointSlice records the backend's eligibility for the Service.

Level 07/08Interface

Follow one Pod through the audit log

Use the timeline to follow web-6d5c46b4dc-5qkld through seven API writes. Each entry shows a captured audit identity, Kubernetes API verb, response code and selected fields. Relative times use the first Deployment create response as zero and have millisecond precision. They measure when the API server logged each response, so the gaps can include queueing, runtime work and the lab's deliberate pause before creating the readiness file. Kubernetes auditing

  1. Apply
  2. ReplicaSet
  3. Pod
  4. Bind
  5. Running
  6. Ready
  7. Endpoint

The API server records the Deployment

API response
+0 ms
Actor
system:admin
Operation
CREATE /apis/apps/v1/namespaces/pod-dive/deployments 201
{
  "name": "web",
  "uid": "3fe3a7e6-9c45-4a9b-bec8-44f87f714c13"
}

The Deployment controller creates a ReplicaSet

API response
+3,768 ms
Actor
system:serviceaccount:kube-system:deployment-controller
Operation
CREATE /apis/apps/v1/namespaces/pod-dive/replicasets 201
{
  "name": "web-6d5c46b4dc",
  "uid": "9a867ace-fc55-4ab4-a939-322490b7dcdb",
  "owner": {
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    "name": "web",
    "uid": "3fe3a7e6-9c45-4a9b-bec8-44f87f714c13",
    "controller": true,
    "blockOwnerDeletion": true
  }
}

The ReplicaSet controller creates a Pod

API response
+3,872 ms
Actor
system:serviceaccount:kube-system:replicaset-controller
Operation
CREATE /api/v1/namespaces/pod-dive/pods 201
{
  "name": "web-6d5c46b4dc-5qkld",
  "uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
  "owner": {
    "apiVersion": "apps/v1",
    "kind": "ReplicaSet",
    "name": "web-6d5c46b4dc",
    "uid": "9a867ace-fc55-4ab4-a939-322490b7dcdb",
    "controller": true,
    "blockOwnerDeletion": true
  }
}

The scheduler commits a node choice

API response
+3,884 ms
Actor
system:kube-scheduler
Operation
CREATE /api/v1/namespaces/pod-dive/pods/web-6d5c46b4dc-5qkld/binding 201
{
  "uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
  "node": "pod-dive-node"
}

The kubelet reports a running container

API response
+19,186 ms
Actor
system:node:pod-dive-node
Operation
PATCH /api/v1/namespaces/pod-dive/pods/web-6d5c46b4dc-5qkld/status 200
{
  "uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
  "node": "pod-dive-node",
  "phase": "Running",
  "podIP": "10.42.0.2",
  "scheduled": true,
  "sandbox": true,
  "ready": false,
  "waiting": null,
  "startedAt": "2026-11-19T00:22:26Z",
  "restarts": 0
}

The kubelet reports the Pod ready

API response
+22,185 ms
Actor
system:node:pod-dive-node
Operation
PATCH /api/v1/namespaces/pod-dive/pods/web-6d5c46b4dc-5qkld/status 200
{
  "uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
  "node": "pod-dive-node",
  "phase": "Running",
  "podIP": "10.42.0.2",
  "scheduled": true,
  "sandbox": true,
  "ready": true,
  "waiting": null,
  "startedAt": "2026-11-19T00:22:26Z",
  "restarts": 0
}

The EndpointSlice controller marks the address ready

API response
+22,198 ms
Actor
system:serviceaccount:kube-system:endpointslice-controller
Operation
UPDATE /apis/discovery.k8s.io/v1/namespaces/pod-dive/endpointslices/web-qsrw8 200
{
  "slice": "web-qsrw8",
  "podUID": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
  "address": "10.42.0.2",
  "ready": true
}
  1. A client-side apply creates a missing object. The response gives the Deployment its UID.
1 / 7
API audit responses from one Kubernetes v1.34.3+k3s1 run. The last step names the EndpointSlice that changed after the Pod became ready.
Download the files and reproduce the capture

Save workload.yaml, audit.yaml and capture.py together in a scratch directory. The collector runs once per fresh container. The captured JSON contains the projected audit records, endpoint snapshots and Pod Events used here.

The capture host had unusually low free disk space. Its disposable node also used --kubelet-arg='eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%,imagefs.inodesFree<1%' on the docker run command. The recipe below uses the normal thresholds for a host with sufficient space. Node pressure eviction

Start a scratch cluster and collect the audit trace
# Run from a scratch directory containing audit.yaml, workload.yaml and capture.py.
docker pull nginx:1.27-alpine
lab="undefinedstack-pod-$(date +%s)"
docker run --rm -d --privileged --name "$lab" --hostname pod-dive-node \
  --label dev.undefinedstack.capture=pod-startup \
  -v "$(pwd)/audit.yaml:/audit.yaml:ro" rancher/k3s:v1.34.3-k3s1 server \
  --disable=traefik,servicelb,metrics-server,local-storage,coredns \
  --kube-apiserver-arg=audit-policy-file=/audit.yaml \
  --kube-apiserver-arg=audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log

k() {
  docker exec -i "$lab" kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pod-dive "$@"
}
# Retry these checks if the API or node is still starting.
k get --raw=/readyz
k wait node/pod-dive-node --for=condition=Ready --timeout=60s

docker save nginx:1.27-alpine | docker exec -i "$lab" ctr \
  --address /run/k3s/containerd/containerd.sock --namespace k8s.io images import -
python3 capture.py "$lab" > capture.json
Inspect the lab and stop its container when finished
k get deployment,replicaset,pod,service,endpointslice -o wide
k get events --sort-by=.metadata.creationTimestamp
docker stop "$lab"

Use a Pod UID to correlate controller ownership, placement, kubelet status and Service endpoint records.

Level 08/08Interface

Find the handoff that stalled

Start with the newest Pod belonging to the affected ReplicaSet. Inspect its node assignment, container state and conditions. Then read Events for that Pod. The three runs below each change one part of the lab manifest. They illustrate why Pending alone can send you looking at the wrong component.

The scheduler cannot find a fit

The container requests 10,000 CPU cores. The lab's only node cannot fit that request.

Observed fields
Pod: no-fit-69449c6b74-thwtf
spec.nodeName: (unset)
status.phase: Pending
status.conditions[PodScheduled]: False or absent
status.conditions[Ready]: False or absent
Pod Event excerpts
FailedScheduling  0/1 nodes are available: 1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Inspect the Pod while the lab is running
k describe pod -l app=no-fit

Read the FailedScheduling message and compare the Pod's requests with eligible nodes' allocatable capacity and already scheduled requests. Adjust the workload's request if it is wrong, or provide a node that can accommodate it.

The runtime cannot retrieve the image

The image refers to a registry at 127.0.0.1:1 inside the lab node. There is no registry listening there.

Observed fields
Pod: bad-image-d7c8c8d8b-fsz8x
spec.nodeName: pod-dive-node
status.phase: Pending
status.conditions[PodScheduled]: True
status.conditions[Ready]: False or absent
container state.waiting.reason: ImagePullBackOff
container restartCount: 0
Pod Event excerpts
Pulling  Pulling image "127.0.0.1:1/undefinedstack/absent:v1"
Failed  Failed to pull image "127.0.0.1:1/undefinedstack/absent:v1": failed to pull and unpack image "127.0.0.1:1/undefinedstack/absent:v1": failed to resolve reference "127.0.0.1:1/undefinedstack/absent:v1": failed to do request: Head "https://127.0.0.1:1/v2/undefinedstack/absent/manifests/v1": dial tcp 127.0.0.1:1: connect: connection refused
Failed  Error: ErrImagePull
BackOff  Back-off pulling image "127.0.0.1:1/undefinedstack/absent:v1"
Failed  Error: ImagePullBackOff
Inspect the Pod while the lab is running
k describe pod -l app=bad-image

Read the first Failed event for the pull error. Check repository and tag, node-to-registry connectivity and pull credentials as the message warrants. ImagePullBackOff describes the retry delay.

The HTTP readiness probe keeps failing

NGINX runs normally, but the probe targets /missing.html. It gets HTTP 404.

Observed fields
Pod: bad-probe-58768c856d-ddx5p
spec.nodeName: pod-dive-node
status.phase: Running
status.conditions[PodScheduled]: True
status.conditions[Ready]: False or absent
container restartCount: 0
EndpointSlice address: 10.42.0.4
EndpointSlice conditions.ready: false
Pod Event excerpts
Started  Started container nginx
Unhealthy  Readiness probe failed: HTTP probe failed with statuscode: 404
Inspect the Pod while the lab is running
k describe pod -l app=bad-probe

Compare the configured probe's path, port and scheme with the application. Check its startup and request logs. Here the endpoint exists with ready=false and the container has zero restarts.

Each tab shows a distinct live-cluster failure. The Pod names were generated during the capture and will differ on a new run.

Events are time-limited and may be aggregated or absent. For an ongoing problem, combine them with current object status, the controller's logs when available and an application check. If a Pod is already ready, compare its labels to the Service selector and inspect the EndpointSlices identified by kubernetes.io/service-name. If a matching endpoint is ready too, test the traffic path from the affected client. Debug Pods · Debug Services

Check yourselfThe Pod has a node and is Running. Its HTTP readiness probe gets 404, with zero restarts. What should you inspect first?

Read the last recorded handoff before choosing where to investigate.

End of Kubernetes Pod startup · 8 levels, Idea to System

Where to next

Browse all 39 divesAll topics