The API server records the Deployment
- API response
- +0 ms
- Actor
- system:admin
- Operation
- CREATE /apis/apps/v1/namespaces/pod-dive/deployments 201
{
"name": "web",
"uid": "3fe3a7e6-9c45-4a9b-bec8-44f87f714c13"
}From kubectl apply to a ready Pod
Infrastructure and operationsDive 01 of 01
Trace one Deployment through the API server, controllers, scheduler and kubelet. Follow a real Pod into a Service and diagnose three places startup can stall.
You run kubectl apply and see deployment.apps/web created. The Pod eventually reports Running while the Service still has zero ready backends. Those observations refer to different state changes, made by different components. Which one is your request waiting for?
Follow one Deployment named web. Its selector identifies the workload and its Pod template describes what to instantiate. The replica count is one. The Service is a separate object with its own selector, app: web. It maps port 80 to the named container port http. The HTTP readiness probe asks for /ready.html, which the stock NGINX image lacks. The lab creates that file after observing the first failure. Deployments · Services
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:1.27-alpine
ports:
- name: http
containerPort: 80
readinessProbe:
httpGet:
path: /ready.html
port: http
periodSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web
ports:
- port: 80
targetPort: httpdeployment.apps/web created
service/web createdThe Deployment declares a desired set of Pods. The Service independently selects Pods that can serve its traffic.
In this run, kubectl used client-side apply. It decoded the manifest, found the API resource for each object and looked up its live state. With no Deployment of that name it sent a create request and stored the configuration in the kubectl.kubernetes.io/last-applied-configuration annotation. On later changes, it calculates a patch using the manifest, the live object and that previous configuration. With --server-side, the server handles the apply merge and records field ownership in managedFields. Client-side merge · Server-Side Apply
The API server authenticates the caller, authorizes the write, and runs admission and API validation before persistence. Defaults can fill unset fields, and admission can change or reject the request. The Deployment and Service are separate writes. A later failure applying the Service leaves the first object in place. Kubernetes commonly uses etcd for API storage. This K3s cluster used its default SQLite datastore behind Kine, which provides an etcd-compatible API. Admission controllers · K3s datastore options · Kine
POST /apis/apps/v1/namespaces/pod-dive/deployments
HTTP 201
metadata.uid = 3fe3a7e6-9c45-4a9b-bec8-44f87f714c13The create response acknowledges the stored Deployment. Workload controllers can act on it independently of the client.
Controllers use the API to observe changes and work toward the state described by each object. The Deployment controller finds a ReplicaSet for the current Pod template or creates one. It adds the pod-template-hash label to the ReplicaSet selector and its Pod template, so ReplicaSets for different revisions select distinct Pods. Changing the Pod template triggers a rollout. Changing only the replica count scales the applicable ReplicaSets without producing another template revision. Deployment rollouts
With a desired count of one and zero matching controlled Pods, the ReplicaSet controller creates a Pod from its template. Each object records its controlling owner through metadata.ownerReferences. In the trace you can follow the UID in the Pod's owner reference to the ReplicaSet, then follow the ReplicaSet's owner UID to the Deployment. A label is a selection rule. The owner UID identifies a particular object even if another object later uses the same name. ReplicaSet reconciliation · Owners and dependents
web
Deployment UID 3fe3a7e6-9c45-4a9b-bec8-44f87f714c13
└─ web-6d5c46b4dc
ReplicaSet UID 9a867ace-fc55-4ab4-a939-322490b7dcdb
owner UID 3fe3a7e6-9c45-4a9b-bec8-44f87f714c13
└─ web-6d5c46b4dc-5qkld
Pod UID c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904
owner UID 9a867ace-fc55-4ab4-a939-322490b7dcdbThe controller watches API objects and queues keys for reconciliation. A watch notification tells the controller what to revisit. Its next pass evaluates the current state it can observe, so repeated notifications can lead to a single pass. If this Pod disappears and the ReplicaSet still wants one replica, it creates a replacement with a different UID. A container restart inside a surviving Pod keeps that Pod's identity. Controller loops · ReplicaSet controller source · Pod lifetime
Deployment → ReplicaSet → Pod is a chain of independently reconciled API objects, with owner references you can inspect.
The fresh Pod has no spec.nodeName. The scheduler pulls an eligible, unassigned Pod from its queue and filters out nodes that fail its constraints. Depending on the Pod, those include resource requests, selectors, required affinity, taints and storage topology. The scheduler scores feasible nodes when there is a choice. CPU fit uses requested resources against allocatable capacity and already assigned requests. Low CPU usage in a metrics dashboard does not make a request fit. Scheduling and binding · Resource requests and scheduling
In our single-node cluster, pod-dive-node passed the filters. Through the Pod's binding subresource, the scheduler submitted a target node and the Pod UID. The API server recorded the assignment as spec.nodeName. That stored field is the handoff to the node agent. If the scheduler cannot place the Pod, the assignment remains empty and you can inspect PodScheduled and the scheduler's Events. Default binder source
POST /api/v1/namespaces/pod-dive/pods/web-6d5c46b4dc-5qkld/binding
HTTP 201
metadata.uid = c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904
target.name = pod-dive-nodeScheduling picks a node. The stored binding assigns the Pod to that node so its kubelet can act.
The kubelet sees the Pod assigned to its node and works to make the actual containers match the spec. It handles prerequisites such as required volumes, then asks the container runtime through the The gRPC API through which the kubelet manages Pod sandboxes, containers and images. to create a Pod sandbox. In this setup, containerd runs the sandbox and invokes the CNI networking plugin. The sandbox supplies the shared network namespace and Pod IP used by containers in that Pod. In Kubernetes 1.34, PodReadyToStartContainers reports sandbox and network readiness and is enabled by default. Container Runtime Interface · Pod network readiness
The kubelet ensures the image is available under its pull policy, then requests container creation and start. This Pod has no init containers and our NGINX image was preloaded. The Events show it is already present, then that the container was created and started. The Pod status patch reports Running, IP 10.42.0.2 and Ready: False. An HTTP 404 is enough to show a process answering on the probe port. It also shows why the readiness probe has yet to pass. Image pull policy · Probe results
Pulled Container image "nginx:1.27-alpine" already present on machine
Created Created container: nginx
Started Started container nginx
Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 404Running is a coarse Pod phase. Per-container Waiting, Running and Terminated states tell you what particular processes are doing. The STATUS column in kubectl get pods can show display reasons such as ContainerCreating or ImagePullBackOff. To distinguish them from the phase and readiness conditions, inspect the object. Pod phases and container states
A node assignment, a prepared sandbox, a running container and a ready Pod represent different milestones.
The kubelet's HTTP probe goes to the Pod IP on the configured port. It bypasses the Service's virtual IP. In the lab, writing /usr/share/nginx/html/ready.html makes that probe pass. A failed readiness probe changes readiness without restarting the process. A liveness failure can trigger a restart. A startup probe, when configured, holds off readiness and liveness checks until startup succeeds. At the Pod level, any configured readiness gates also need to be true. Readiness, liveness and startup probes · Pod readiness gates
The EndpointSlice controller watches matching Pods and publishes endpoint addresses and conditions for the Service. Our slice already held the Pod's address with ready: false. After the kubelet reported Ready=True, the controller updated that same endpoint to ready: true. Consumers such as kube-proxy watch the slices and update their forwarding rules. In this lab, an HTTP request through the Service ClusterIP returned the file after the endpoint became ready. The API write and local forwarding update are separate steps. The Service option publishNotReadyAddresses can deliberately publish endpoints as ready regardless of Pod readiness. EndpointSlice conditions · Service virtual IPs
same EndpointSlice web-qsrw8
same endpoint IP 10.42.0.2
same target UID c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904
conditions.ready false → true
GET http://10.43.178.85/ready.html
readyDeployment availability is reported separately. A newly ready Pod can already be a Service backend while the Deployment still waits out its minReadySeconds before counting it as available. That value defaults to zero in our manifest. Deployment availability timing
Pod readiness is kubelet status. An EndpointSlice records the backend's eligibility for the Service.
Use the timeline to follow web-6d5c46b4dc-5qkld through seven API writes. Each entry shows a captured audit identity, Kubernetes API verb, response code and selected fields. Relative times use the first Deployment create response as zero and have millisecond precision. They measure when the API server logged each response, so the gaps can include queueing, runtime work and the lab's deliberate pause before creating the readiness file. Kubernetes auditing
{
"name": "web",
"uid": "3fe3a7e6-9c45-4a9b-bec8-44f87f714c13"
}{
"name": "web-6d5c46b4dc",
"uid": "9a867ace-fc55-4ab4-a939-322490b7dcdb",
"owner": {
"apiVersion": "apps/v1",
"kind": "Deployment",
"name": "web",
"uid": "3fe3a7e6-9c45-4a9b-bec8-44f87f714c13",
"controller": true,
"blockOwnerDeletion": true
}
}{
"name": "web-6d5c46b4dc-5qkld",
"uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
"owner": {
"apiVersion": "apps/v1",
"kind": "ReplicaSet",
"name": "web-6d5c46b4dc",
"uid": "9a867ace-fc55-4ab4-a939-322490b7dcdb",
"controller": true,
"blockOwnerDeletion": true
}
}{
"uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
"node": "pod-dive-node"
}{
"uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
"node": "pod-dive-node",
"phase": "Running",
"podIP": "10.42.0.2",
"scheduled": true,
"sandbox": true,
"ready": false,
"waiting": null,
"startedAt": "2026-11-19T00:22:26Z",
"restarts": 0
}{
"uid": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
"node": "pod-dive-node",
"phase": "Running",
"podIP": "10.42.0.2",
"scheduled": true,
"sandbox": true,
"ready": true,
"waiting": null,
"startedAt": "2026-11-19T00:22:26Z",
"restarts": 0
}{
"slice": "web-qsrw8",
"podUID": "c1d47a5f-c8f6-4b6e-965e-63d5cc3c2904",
"address": "10.42.0.2",
"ready": true
}Save workload.yaml, audit.yaml and capture.py together in a scratch directory. The collector runs once per fresh container. The captured JSON contains the projected audit records, endpoint snapshots and Pod Events used here.
The capture host had unusually low free disk space. Its disposable node also used --kubelet-arg='eviction-hard=memory.available<100Mi,nodefs.available<1%,imagefs.available<1%,nodefs.inodesFree<1%,imagefs.inodesFree<1%' on the docker run command. The recipe below uses the normal thresholds for a host with sufficient space. Node pressure eviction
# Run from a scratch directory containing audit.yaml, workload.yaml and capture.py.
docker pull nginx:1.27-alpine
lab="undefinedstack-pod-$(date +%s)"
docker run --rm -d --privileged --name "$lab" --hostname pod-dive-node \
--label dev.undefinedstack.capture=pod-startup \
-v "$(pwd)/audit.yaml:/audit.yaml:ro" rancher/k3s:v1.34.3-k3s1 server \
--disable=traefik,servicelb,metrics-server,local-storage,coredns \
--kube-apiserver-arg=audit-policy-file=/audit.yaml \
--kube-apiserver-arg=audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log
k() {
docker exec -i "$lab" kubectl --kubeconfig=/etc/rancher/k3s/k3s.yaml -n pod-dive "$@"
}
# Retry these checks if the API or node is still starting.
k get --raw=/readyz
k wait node/pod-dive-node --for=condition=Ready --timeout=60s
docker save nginx:1.27-alpine | docker exec -i "$lab" ctr \
--address /run/k3s/containerd/containerd.sock --namespace k8s.io images import -
python3 capture.py "$lab" > capture.jsonk get deployment,replicaset,pod,service,endpointslice -o wide
k get events --sort-by=.metadata.creationTimestamp
docker stop "$lab"Use a Pod UID to correlate controller ownership, placement, kubelet status and Service endpoint records.
Start with the newest Pod belonging to the affected ReplicaSet. Inspect its node assignment, container state and conditions. Then read Events for that Pod. The three runs below each change one part of the lab manifest. They illustrate why Pending alone can send you looking at the wrong component.
The container requests 10,000 CPU cores. The lab's only node cannot fit that request.
Pod: no-fit-69449c6b74-thwtf
spec.nodeName: (unset)
status.phase: Pending
status.conditions[PodScheduled]: False or absent
status.conditions[Ready]: False or absentFailedScheduling 0/1 nodes are available: 1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.k describe pod -l app=no-fitRead the FailedScheduling message and compare the Pod's requests with eligible nodes' allocatable capacity and already scheduled requests. Adjust the workload's request if it is wrong, or provide a node that can accommodate it.
The image refers to a registry at 127.0.0.1:1 inside the lab node. There is no registry listening there.
Pod: bad-image-d7c8c8d8b-fsz8x
spec.nodeName: pod-dive-node
status.phase: Pending
status.conditions[PodScheduled]: True
status.conditions[Ready]: False or absent
container state.waiting.reason: ImagePullBackOff
container restartCount: 0Pulling Pulling image "127.0.0.1:1/undefinedstack/absent:v1"
Failed Failed to pull image "127.0.0.1:1/undefinedstack/absent:v1": failed to pull and unpack image "127.0.0.1:1/undefinedstack/absent:v1": failed to resolve reference "127.0.0.1:1/undefinedstack/absent:v1": failed to do request: Head "https://127.0.0.1:1/v2/undefinedstack/absent/manifests/v1": dial tcp 127.0.0.1:1: connect: connection refused
Failed Error: ErrImagePull
BackOff Back-off pulling image "127.0.0.1:1/undefinedstack/absent:v1"
Failed Error: ImagePullBackOffk describe pod -l app=bad-imageRead the first Failed event for the pull error. Check repository and tag, node-to-registry connectivity and pull credentials as the message warrants. ImagePullBackOff describes the retry delay.
NGINX runs normally, but the probe targets /missing.html. It gets HTTP 404.
Pod: bad-probe-58768c856d-ddx5p
spec.nodeName: pod-dive-node
status.phase: Running
status.conditions[PodScheduled]: True
status.conditions[Ready]: False or absent
container restartCount: 0
EndpointSlice address: 10.42.0.4
EndpointSlice conditions.ready: falseStarted Started container nginx
Unhealthy Readiness probe failed: HTTP probe failed with statuscode: 404k describe pod -l app=bad-probeCompare the configured probe's path, port and scheme with the application. Check its startup and request logs. Here the endpoint exists with ready=false and the container has zero restarts.
Events are time-limited and may be aggregated or absent. For an ongoing problem, combine them with current object status, the controller's logs when available and an application check. If a Pod is already ready, compare its labels to the Service selector and inspect the EndpointSlices identified by kubernetes.io/service-name. If a matching endpoint is ready too, test the traffic path from the affected client. Debug Pods · Debug Services
Read the last recorded handoff before choosing where to investigate.
End of Kubernetes Pod startup · 8 levels, Idea to System
New path · Application development
That was the last dive in Infrastructure and operations. The next path starts here.
Follow a recent-orders query from cardinality estimates to a B-tree leaf and the heap. Change the data and index, inspect actual plans and see why an index-only scan can still fetch a row.