Control Plane, Worker Nodes & Internal Cluster Communication (Complete Practical Guide)

Unfortunately, most engineers skip Kubernetes architecture and jump directly into Deployments, Helm, or CI/CD pipelines. However, this usually becomes a serious problem later when production issues appear.

Before understanding Kubernetes architecture, it is important to understand Kubernetes core objects such as Pods, Deployments, and Services.

When clusters fail, Pods remain pending, or networking breaks unexpectedly, engineers who do not understand architecture often struggle to troubleshoot effectively.

Therefore, before learning advanced Kubernetes topics, you must deeply understand how Kubernetes itself works internally.

In this guide, you will learn:

How Kubernetes architecture is designed
How control plane components interact
How worker nodes execute workloads
How scheduling actually happens
How etcd stores cluster state
How kubelet and kube-proxy work internally
How requests travel through the cluster
Real troubleshooting workflows
Enterprise architecture best practices

This is not just theory. We will inspect real components, logs, APIs, metrics, and scheduler decisions step by step.

Who This Guide Is For

This guide is designed for:

Beginners learning Kubernetes internals
DevOps engineers preparing for production operations
Platform engineers troubleshooting clusters
Engineers preparing for CKA/CKS exams
Anyone confused about how Kubernetes works behind the scenes

Why Kubernetes Architecture Matters

Most Kubernetes problems are actually architecture problems.

For deeper technical details, refer to the official Kubernetes architecture documentation.

For example:

Problem	Real Cause
Pod stuck Pending	Scheduler issue
API commands fail	API Server unavailable
Cluster state inconsistent	etcd issue
Networking broken	kube-proxy or CNI issue
Node NotReady	kubelet issue

Without understanding architecture:

Troubleshooting becomes guesswork
Debugging becomes slow
Production incidents become dangerous

Architecture knowledge transforms Kubernetes from:

“magic platform”

into:

“predictable distributed system”

High-Level Kubernetes Architecture

A Kubernetes cluster consists of two major parts:

Component	Purpose
Control Plane	Brain of the cluster
Worker Nodes	Execute workloads

The Control Plane (The Brain)

The Control Plane is responsible for:

Making decisions
Maintaining cluster state
Scheduling workloads
Running controllers
Exposing APIs

Without the Control Plane:

Kubernetes cannot function

Kubernetes Architecture Overview

The Ray-Ban Meta Wayfarer Smart Glasses combine iconic style with smart features designed for modern lifestyles. A great choice for creators, professionals, and tech enthusiasts who want hands-free convenience, smart connectivity, and everyday comfort in a stylish form factor. This post may contain affiliate links. Buy now

Buy Now

Core Control Plane Components

The Kubernetes Control Plane is the brain of the cluster. It is responsible for making decisions, maintaining the cluster’s desired state, and ensuring workloads run correctly across worker nodes.

The Control Plane contains several critical services that work together continuously.

API Server — The Entry Point

The API Server is the most important component in Kubernetes.

Everything communicates through it.

What the API Server Does

At a high level, the API Server is responsible for:

Receiving all cluster requests
Validating Kubernetes objects
Authenticating users and services
Authorizing permissions using RBAC
Updating cluster state inside etcd
Exposing REST APIs
Coordinating communication between components

The API Server is the heart of the Kubernetes Control Plane. It acts as the central communication hub for the entire cluster.

Every action in Kubernetes — whether it comes from a user, controller, scheduler, kubelet, or external automation tool — must first go through the API Server.

Because of this, the API Server is often described as:

“The front door of Kubernetes.”

Without the API Server, Kubernetes components cannot communicate with each other, cluster state cannot be updated, and workloads cannot be managed properly.

It acts as the central layer between:

Users
Kubernetes components
Cluster state

You never talk directly to:

Pods
Nodes
Controllers

Everything flows through:

kube-apiserver

Why Kubernetes Needs an API Server

Kubernetes is a distributed system with many independent components running simultaneously.

For example:

Controllers continuously monitor workloads
kubelets report node status
Schedulers assign Pods to nodes
Users submit deployment requests
CI/CD pipelines update applications

If every component directly communicated with every other component:

Complexity would become unmanageable
State consistency would break
Security would become difficult

Instead, Kubernetes centralizes communication through:

kube-apiserver

This creates:

Standardized communication
Secure access control
Consistent cluster state
Reliable orchestration

Example Request Flow

Suppose you run:

kubectl apply -f deployment.yaml

This happens internally:

Step 1 — kubectl Sends Request

kubectl converts YAML into an API request.

The request is sent to:

kube-apiserver

Step 2 — Authentication

The API Server verifies:

Who is making the request
Whether credentials are valid

This may involve:

Certificates
Tokens
Azure AD
IAM integrations

Step 3 — Authorization

After authentication, Kubernetes checks permissions using RBAC.

Example:

Can this user create Deployments?
Can this service account delete Pods?

If permissions fail:

Request is rejected

Step 4 — Validation

The API Server validates:

YAML structure
API version
Required fields
Schema correctness

For example:

apiVersion: apps/v1
kind: Deployment

If fields are invalid:

Request fails immediately

Step 5 — Store State in etcd

Once validated:

Desired state is stored inside etcd

At this point:

Kubernetes now knows what you WANT

However:

Nothing has been scheduled yet

Step 6 — Controllers Detect Changes

Controllers continuously watch the API Server.

The Deployment Controller notices:

New Deployment created

It then creates:

ReplicaSets
Pod definitions

Step 7 — Scheduler Assigns Nodes

The Scheduler detects unscheduled Pods.

It decides:

Which worker node should run each Pod

Step 8 — kubelet Starts Containers

The worker node kubelet receives Pod instructions.

It:

Pulls container images
Starts containers
Reports status back

This entire workflow begins at:

API Server

Why Everything Uses the API Server

The API Server acts as:

Single source of truth access layer
Central coordination engine
Security enforcement point

Even internal Kubernetes components do not bypass it.

For example:

Component	Communicates Through API Server?
Scheduler	Yes
Controller Manager	Yes
kubelet	Yes
kubectl	Yes
Argo CD	Yes
Helm	Yes

This design ensures:

Consistency
Auditability
Security
Reliability

API Server Security Responsibilities

The API Server is also responsible for cluster security.

It handles:

Security Function	Purpose
Authentication	Verify identity
Authorization	Check permissions
Admission Controllers	Enforce policies
Audit Logging	Record activity

This is why production Kubernetes security heavily focuses on:

API Server hardening

Admission Controllers

Admission Controllers intercept requests before objects are stored.

They can:

Modify requests
Reject invalid configurations
Enforce security policies

Example:

Prevent privileged containers
Enforce labels
Block unsafe images

This is critical in enterprise environments.

Kubernetes APIs

The API Server exposes REST APIs.

Example endpoints:

/api
/apis
/api/v1/pods
/apis/apps/v1/deployments

You can query them directly:

kubectl get --raw /api

Practical Lab — Inspect API Server

Step 1: View Cluster Info

kubectl cluster-info

This shows:

API Server endpoint
Core cluster services

Step 2: View API Resources

kubectl api-resources

This displays:

Supported Kubernetes resource types

View API Versions

kubectl api-versions

Step 3: Access Health Endpoint

kubectl get --raw='/healthz'

Expected:

ok

Step 4: View API Metrics

kubectl get --raw='/metrics'

This exposes:

Request latency
API throughput
Error counts
Cluster performance metrics

Monitoring tools like:

Prometheus
Grafana
Datadog

often collect these metrics.

Enterprise monitoring tools use these metrics heavily.

What Happens If API Server Fails?

If the API Server becomes unavailable:

kubectl stops working
Controllers cannot reconcile state
Scheduler cannot assign Pods
kubelets cannot update status
Cluster operations freeze

Already running containers may continue temporarily, but:

Kubernetes loses orchestration capability

This is why production clusters use:

Highly Available API Servers

etcd — The Cluster Database

etcd is one of the most important components in Kubernetes. It acts as the central database of the entire cluster and stores all Kubernetes data reliably.

At its core, etcd is a distributed key-value store designed for:

High availability
Strong consistency
Fault tolerance
Fast reads and writes

Kubernetes depends heavily on etcd because it stores the entire cluster’s desired and current state.

This is why etcd is often called:

“The memory of Kubernetes.”

Without etcd, Kubernetes cannot remember:

What applications should run
Which Pods exist
What Services are configured
What the cluster should look like

Without etcd:

Kubernetes loses memory

What Does etcd Actually Store?

etcd stores almost everything Kubernetes needs to operate.

This includes:

Resource	Example
Deployments	Replica count, image versions
Pods	Pod specifications
Services	Networking definitions
ConfigMaps	Application configuration
Secrets	Passwords, certificates
Namespaces	Logical isolation
Nodes	Worker node information
RBAC Policies	Permissions
Cluster State	Entire desired state

Whenever you create an object in Kubernetes:

The API Server stores it inside etcd

Suppose you run:

kubectl apply -f deployment.yaml

Many engineers think Kubernetes immediately creates containers.

However, the actual workflow is:

Step 1 — YAML Sent to API Server

kubectl sends the Deployment request to:

kube-apiserver

Step 2 — API Server Validates Request

The API Server checks:

Syntax
Authentication
Authorization
API schema

Step 3 — Object Stored in etcd

After validation:

Deployment definition is stored in etcd

At this stage:

Kubernetes now remembers the desired state

Example stored data:

replicas: 3
image: nginx:1.25

Step 4 — Controllers React

Controllers continuously watch for changes through the API Server.

The Deployment Controller notices:

Desired replicas = 3
Actual replicas = 0

Then Kubernetes starts creating Pods.

Desired State vs Actual State

This is one of the most important Kubernetes concepts.

Desired State

Desired state means:

What you WANT the cluster to look like

Example:

replicas: 3

This means:

I want 3 Pods running at all times

This desired state gets stored inside:

etcd

Actual State

Actual state means:

What is currently happening inside the cluster

Example:

2 Pods running
1 Pod crashed

Controllers compare:

Desired state (etcd)
Actual cluster state

If they differ:

Kubernetes fixes the mismatch

This entire system depends on etcd storing reliable cluster state.

Why etcd Is So Critical

Without etcd:

Kubernetes cannot remember cluster configuration
Controllers lose desired state
Scheduling decisions fail
API operations become inconsistent

etcd is effectively:

The source of truth for Kubernetes

Everything depends on it.

What Happens If etcd Fails?

If etcd becomes corrupted or unavailable, Kubernetes experiences severe problems.

Possible Failures

Cluster State Loss

Kubernetes may lose:

Deployments
Services
Configurations
Secrets

Scheduling Stops

The Scheduler cannot determine:

Existing Pods
Desired replicas
Cluster resource state

API Server Becomes Unstable

The API Server depends on etcd for:

Reading state
Writing updates

Without etcd:

API requests fail

Controllers Stop Reconciling

Controllers cannot compare:

Desired state
Actual state

Self-healing breaks.

Why etcd Uses Distributed Consensus

etcd is distributed across multiple nodes for reliability.

It uses:

Raft Consensus Algorithm

This ensures:

Data consistency
Leader election
Reliable synchronization

Even if one node fails:

etcd can continue functioning

This is essential in production Kubernetes clusters.

etcd Architecture

A production etcd cluster usually contains:

Node	Role
Leader	Handles writes
Followers	Replicate data

All changes must be agreed upon through consensus.

This prevents:

Inconsistent cluster state

Split-brain scenarios

Practical Lab — Explore etcd Objects

Step 1: Create ConfigMap

kubectl create configmap demo-config --from-literal=env=prod

Step 2: Retrieve Object

kubectl get configmap demo-config -o yaml

This object exists inside etcd.

This object exists because:

API Server stored it inside etcd

Step 3: Understand State Persistence

Delete Pod:

kubectl delete pod <pod-name>

Kubernetes recreates it because:

Desired state still exists in etcd

Practical Lab — Observe Kubernetes State Storage

Step 1 — Create ConfigMap

kubectl create configmap demo-config --from-literal=env=prod

Step 2 — Retrieve ConfigMap

kubectl get configmap demo-config -o yaml

This object exists because:

API Server stored it inside etcd

Step 3 — Modify State

Update ConfigMap:

kubectl edit configmap demo-config

Kubernetes immediately updates:

etcd cluster state

Practical Lab — Understand Desired State

Step 1 — Create Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx

Apply:

kubectl apply -f deployment.yaml

Step 2 — Delete a Pod

kubectl delete pod <pod-name>

Step 3 — Observe Recovery

kubectl get pods -w

Kubernetes recreates the Pod because:

Desired state still exists in etcd

etcd Security

etcd stores extremely sensitive information:

Secrets
Tokens
Certificates
Cluster credentials

Therefore, production etcd must be secured carefully.

Enterprise Security Best Practices

✅ Enable TLS encryption
✅ Restrict etcd access
✅ Encrypt Secrets at rest
✅ Use network isolation
✅ Monitor etcd health
✅ Rotate certificates regularly

Why etcd Backups Are Critical

If etcd data becomes corrupted:

Entire clusters may become unrecoverable

This is why production environments always:

Backup etcd regularly

Practical Lab — etcd Snapshot Backup

Example backup command:

ETCDCTL_API=3 etcdctl snapshot save snapshot.db

Restore example:

ETCDCTL_API=3 etcdctl snapshot restore snapshot.db

Why Managed Kubernetes Services Hide etcd

Managed platforms like:

usually hide etcd management from users because:

etcd operations are complex
Backups are critical
High availability is difficult

The cloud provider manages:

Replication
Upgrades
Recovery
Maintenance

Common etcd Mistakes

No backups
Single-node etcd clusters
Exposing etcd publicly
Ignoring disk latency
Large object storage abuse

Enterprise Best Practices

Production Kubernetes environments should:

Use multi-node etcd clusters
Monitor etcd latency
Backup frequently
Test disaster recovery
Encrypt communication
Monitor storage usage

Final Thoughts

etcd is not just a database.

It is:

Kubernetes memory
Kubernetes source of truth
The foundation of reconciliation
The backbone of cluster consistency

Every Deployment, Pod, Secret, and Service ultimately depends on:

etcd reliability

Once you understand etcd, you begin understanding:

Why Kubernetes behaves predictably
How reconciliation works
Why backups matter
How cluster state is maintained

Without etcd:

Kubernetes cannot function reliably.

Scheduler — The Decision Maker

The Kubernetes Scheduler is one of the most important components inside the Control Plane. Its primary responsibility is to decide:

Which worker node should run a Pod

Whenever a new Pod is created, Kubernetes does not immediately know where that Pod should run. The Pod first enters a Pending state.

At this stage, the Scheduler analyzes the cluster and determines the most appropriate node for that workload.

Without the Scheduler:

Pods remain Pending forever
Workloads never start
Kubernetes cannot distribute applications across nodes

This is why the Scheduler is often called:

“The brain behind workload placement.”

Scheduler Considers

Factor	Example
CPU availability	Enough resources
Memory	Sufficient RAM
Taints/Tolerations	Node restrictions
Affinity Rules	Placement policies
Node Labels	Workload targeting

Why Scheduling Is Important

In small environments with one node:

Scheduling looks simple

However, enterprise Kubernetes clusters may contain:

Hundreds of nodes
Thousands of Pods
Multiple workloads
Different resource requirements

The Scheduler must intelligently decide:

Where workloads should run
How resources should be distributed
How failures should be handled
How performance should be optimized

Poor scheduling decisions can cause:

Resource starvation
Application instability
Node overload
Performance degradation

Therefore, scheduling is one of the most critical operations in Kubernetes.

What Happens When a Pod Is Created?

Suppose you apply:

kubectl apply -f deployment.yaml

The following internal workflow happens:

Step 1 — API Server Receives Request

The Deployment object is submitted to:

kube-apiserver

Step 2 — Deployment Stored in etcd

Desired state gets stored in:

etcd

At this point:

No containers are running yet

Step 3 — Deployment Controller Creates Pods

The Controller Manager notices:

Desired replicas = 3

It creates Pod objects.

However:

These Pods still have no assigned node

Step 4 — Scheduler Detects Unscheduled Pods

The Scheduler continuously watches the API Server.

It searches for Pods where:

spec.nodeName = empty

These Pods require scheduling.

Step 5 — Scheduler Evaluates Nodes

The Scheduler now:

Scans worker nodes
Applies scheduling logic
Filters unsuitable nodes
Scores remaining nodes

Step 6 — Best Node Selected

After evaluation:

Scheduler chooses optimal node

It updates the Pod object:

spec.nodeName = worker-node-2

Step 7 — kubelet Starts Pod

The kubelet running on that node detects:

A new Pod was assigned to me

Then:

Container image is pulled
Containers start running

Scheduling Is NOT Random

Many beginners think:

“Kubernetes randomly picks nodes.”

This is incorrect.

The Scheduler uses sophisticated algorithms to determine placement.

It evaluates:

Resources
Policies
Constraints
Affinity rules
Node health
Topology awareness

Scheduling is an optimization problem.

How Scheduling Works Internally

Scheduling usually occurs in two phases:

Phase	Purpose
Filtering	Remove unsuitable nodes
Scoring	Rank remaining nodes

Filtering Phase

The Scheduler first eliminates nodes that cannot run the Pod.

Example reasons:

Not enough CPU
Not enough memory
Node taints
Affinity mismatch
Port conflicts

Suppose cluster has 5 nodes:

Node	Result
Node1	Insufficient memory
Node2	Taint mismatch
Node3	Valid
Node4	Valid
Node5	Insufficient CPU

Only:

Node3
Node4

remain candidates.

Scoring Phase

The Scheduler now scores valid nodes.

Factors may include:

Balanced resource usage
Node locality
Existing workloads
Performance optimization

Example:

Node	Score
Node3	72
Node4	91

Scheduler selects:

Node4

Scheduler Considers Many Factors

The Scheduler evaluates many cluster conditions before placement.

CPU Availability

A node must have enough CPU resources.

Example:

resources:
  requests:
    cpu: "500m"

If node lacks CPU:

Scheduling fails

Memory Availability

Memory is another critical factor.

Example:

resources:
  requests:
    memory: "512Mi"

If insufficient memory exists:

Pod remains Pending

Taints & Tolerations

Taints restrict workloads from running on certain nodes.

Example taint:

kubectl taint nodes node1 dedicated=database:NoSchedule

This means:

Pods cannot run there unless they tolerate the taint

Toleration example:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "database"
  effect: "NoSchedule"

Node Labels

Nodes can contain labels:

kubectl label node node1 env=production

Pods may target specific nodes using selectors.

Example:

nodeSelector:
  env: production

This ensures workloads run only on:

Production nodes

Affinity & Anti-Affinity

Affinity rules influence workload placement.

Pod Affinity

Place Pods together.

Example:

Web app near database

Pod Anti-Affinity

Separate Pods across nodes.

Example:

Prevent all replicas running on same node

This improves:

High availability
Fault tolerance

Practical Lab — Observe Scheduler Decisions

Now let’s observe scheduling behavior directly.

Step 1 — Create Deployment

Create file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scheduler-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: nginx
        image: nginx

Save as:

deployment.yaml

Apply:

kubectl apply -f deployment.yaml

Step 2 — Check Pod Placement

Run:

kubectl get pods -o wide

Example output:

NAME                              NODE
scheduler-demo-abc123            worker-node-1
scheduler-demo-def456            worker-node-2
scheduler-demo-ghi789            worker-node-2

You can now observe:

Pod IPs
Assigned nodes
Scheduling distribution

Step 3 — View Scheduler Events

Inspect Pod:

kubectl describe pod <pod-name>

Look for event:

Successfully assigned default/demo-pod to worker-node

This confirms:

Scheduler selected node successfully

Observe Scheduling Decisions in Real Time

Watch events live:

kubectl get events --sort-by=.metadata.creationTimestamp

You can see:

Scheduling events
Failed scheduling attempts
Resource constraints

Why Pods Stay Pending

One of the most common Kubernetes issues is:

Pods stuck in Pending state

This usually means:

Scheduler cannot find suitable node

Common Causes of Pending Pods

Insufficient CPU

Example:

0/3 nodes available: insufficient cpu

Insufficient Memory

Example:

0/3 nodes available: insufficient memory

Taint Mismatch

Pod lacks required toleration.

Affinity Rules Too Strict

Affinity constraints cannot be satisfied.

Node Selector Mismatch

Pod requested labels that no node has.

Practical Lab — Force Scheduling Failure

Create unrealistic CPU request:

resources:
  requests:
    cpu: "100"

Apply Deployment.

Now inspect:

kubectl describe pod <pod-name>

You will see:

FailedScheduling

This is real-world troubleshooting.

Default Scheduler vs Custom Schedulers

Kubernetes includes:

Default Scheduler

However, enterprises may build:

Custom schedulers
AI-aware schedulers
GPU schedulers
Cost-optimized schedulers

This is common in:

ML workloads
Large-scale platforms
HPC environments

Enterprise Scheduling Best Practices

Production clusters should:

Define resource requests
Use anti-affinity for HA
Separate workloads with taints
Monitor scheduling latency
Avoid overcommitting nodes
Use topology-aware scheduling

Common Beginner Mistakes

No resource requests
All workloads on same nodes
Ignoring Pending Pods
Misconfigured affinity rules
Hardcoding node names

FAQs

What does the Scheduler do?

The Scheduler selects the best node for a Pod.

Why are Pods Pending?

Usually because no node satisfies scheduling requirements.

Does the Scheduler start containers?

No. kubelet starts containers after scheduling.

Can Kubernetes schedule Pods automatically?

Yes. Scheduling is fully automated through the Scheduler.

Controller Manager — The Automation Engine

The Controller Manager is one of the most important components inside the Kubernetes Control Plane. It is responsible for running the controllers that continuously monitor and maintain the cluster state.

If the API Server is the communication hub of Kubernetes, then the Controller Manager is:

“The automation engine of Kubernetes.”

It is the component responsible for making Kubernetes:

Self-healing
Declarative
Automated
Reliable

Without the Controller Manager:

Failed Pods would never recover
Replica counts would drift
Nodes would not be monitored
Workloads would become inconsistent

Kubernetes would lose its automation capabilities entirely.

What Is a Controller?

A controller is a control loop that continuously:

Watches cluster state
Compares desired state vs actual state
Takes corrective action if differences exist

This process is called:

Reconciliation

Controllers never stop running.

They continuously work in the background to ensure:

Reality matches your declared configuration

Desired State vs Actual State

This is the foundation of Kubernetes automation.

Desired State

Desired state is:

What you WANT Kubernetes to look like

Example:

replicas: 3

This means:

I want 3 Pods running at all times

Actual State

Actual state is:

What currently exists in the cluster

Example:

Only 2 Pods are running

This difference is called:

Drift

Controllers continuously detect drift and fix it automatically.

Why Controller Manager Exists

In distributed systems:

Containers fail
Nodes crash
Applications stop unexpectedly

Without automation:

Engineers would manually recover everything

This does not scale.

The Controller Manager solves this problem by automating cluster operations continuously.

What Does Controller Manager Actually Run?

The Controller Manager contains multiple controllers.

Each controller is responsible for a specific Kubernetes resource.

Deployment Controller

The Deployment Controller manages:

Deployments
Rolling updates
Rollbacks
ReplicaSets

It ensures:

Correct application versions
Safe updates
Desired replica counts

Example

Desired state:

replicas: 3

Actual state:

2 Pods running

Deployment Controller action:

Create 1 additional Pod

ReplicaSet Controller

ReplicaSet Controller ensures:

Correct number of Pod replicas exist

It continuously watches:

Pod failures
Deleted Pods
Missing replicas

If a Pod disappears:

ReplicaSet Controller creates a replacement automatically

Node Controller

The Node Controller monitors:

Worker node health
Node heartbeats
Node availability

If a node becomes unavailable:

Pods may be rescheduled elsewhere

This improves:

Fault tolerance
Availability

Job Controller

The Job Controller manages:

Batch jobs
One-time tasks
Scheduled workloads

Example:

Database migration
Backup jobs
Report generation

It ensures:

Jobs complete successfully

Controllers Are Continuous Loops

One of the most important concepts in Kubernetes is:

Controllers never execute only once.

Instead:

They continuously monitor cluster state forever

This design enables:

Self-healing
Automatic recovery
Continuous reconciliation

Understanding Reconciliation

Reconciliation is the process where Kubernetes continuously:

Reads desired state
Observes actual state
Fixes differences automatically

This loop never stops.

Real-World Analogy

Think of a thermostat.

Desired state:

22°C room temperature

Actual state:

19°C

Thermostat action:

Turn heater ON

Controllers work exactly the same way.

Kubernetes Reconciliation Flow

The workflow typically looks like this:

Desired State → API Server → etcd
                            ↓
                    Controller Watches
                            ↓
                  Actual State Compared
                            ↓
                  Differences Detected
                            ↓
                    Corrective Action

This is the core automation model of Kubernetes.

Practical Lab — Observe Reconciliation

Now let’s observe reconciliation in action.

Step 1 — Create Deployment

Create file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reconciliation-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: nginx
        image: nginx

Save as:

deployment.yaml

Apply:

kubectl apply -f deployment.yaml

Step 2 — Verify Pods

Run:

kubectl get pods

Expected:

3 Pods running

Step 3 — Delete a Pod

Now intentionally break the system:

kubectl delete pod <pod-name>

Many beginners expect:

Application failure

However, Kubernetes behaves differently.

Step 4 — Watch Reconciliation

Run:

kubectl get pods -w

You will observe:

Deleted Pod disappears
New Pod automatically appears

What Happened Internally?

Here is what occurred behind the scenes:

Pod Deleted

Actual state became:

2 Pods running

ReplicaSet Controller Detects Drift

Desired state:

3 replicas

Actual state:

2 replicas

Controller Takes Action

ReplicaSet Controller creates:

1 replacement Pod

Scheduler Assigns Node

Scheduler selects:

Appropriate worker node

kubelet Starts Container

Node kubelet:

Pulls image
Starts container
Reports status

Why This Is Called Self-Healing

Kubernetes automatically recovered from failure without human intervention.

This behavior is called:

Self-healing

Controllers make self-healing possible.

Important Clarification

Controllers do NOT:

Repair broken application code
Fix container crashes caused by bugs
Solve logic issues

Controllers only ensure:

Desired infrastructure state is maintained

Multiple Controllers Work Together

Controllers cooperate continuously.

Example deployment workflow:

Controller	Responsibility
Deployment Controller	Manages Deployment
ReplicaSet Controller	Maintains replicas
Scheduler	Assigns nodes
kubelet	Starts containers

Kubernetes automation is a collaborative system.

Observe Controllers in Real Time

View Deployments:

kubectl get deployments

View ReplicaSets:

kubectl get rs

View Pods:

kubectl get pods

You can see:

Layered controller relationships

Why Controllers Matter in Production

Enterprise Kubernetes environments rely heavily on controllers for:

Automatic recovery
High availability
Scaling
Rolling updates
Stateful workload management
Cluster consistency

Without controllers:

Kubernetes loses automation

Common Beginner Misunderstandings

Thinking Deployments directly create Pods
Assuming Pods are permanent
Manually recreating failed Pods
Ignoring ReplicaSets
Confusing controllers with schedulers

Enterprise Best Practices

Production clusters should:

Use Deployments instead of raw Pods
Define replica counts carefully
Monitor controller health
Observe reconciliation events
Use GitOps for declarative state
Avoid manual infrastructure drift

FAQs

What does the Controller Manager do?

It runs Kubernetes controllers that maintain cluster state automatically.

What is reconciliation?

Reconciliation is the process of continuously comparing desired and actual state.

Why does Kubernetes recreate deleted Pods?

Because controllers detect missing replicas and restore desired state automatically.

Are controllers always running?

Yes. Controllers continuously monitor the cluster.

Worker Nodes — The Muscle of Kubernetes

If the Control Plane is considered the brain of Kubernetes, then Worker Nodes are the muscle that actually executes workloads.

The Control Plane makes decisions, but worker nodes perform the real execution work.

Every application you deploy in Kubernetes eventually runs on a worker node.

Without worker nodes:

No containers run
No applications execute
No workloads exist

Worker nodes are responsible for:

Running containers
Hosting Pods
Reporting health status
Managing local networking
Communicating with the Control Plane

This is where actual application execution happens.

What Is a Worker Node?

A worker node is a machine — physical or virtual — that participates in the Kubernetes cluster and runs workloads assigned by the Scheduler.

A Kubernetes cluster can contain:

One worker node
Hundreds of worker nodes
Thousands of worker nodes in enterprise environments

Each worker node provides:

CPU
Memory
Storage
Networking resources

for running containers.

Worker Node Architecture

Every worker node typically contains:

Component	Purpose
kubelet	Node agent
kube-proxy	Service networking
Container Runtime	Runs containers
Pods	Application workloads

The architecture generally looks like this:

Control Plane
      ↓
API Server
      ↓
Worker Node
 ├── kubelet
 ├── kube-proxy
 ├── container runtime
 └── Pods

Each component has a specialized responsibility.

kubelet — The Node Agent

kubelet is the most important process running on every worker node.

It acts as the communication bridge between:

Worker node
Kubernetes Control Plane

kubelet continuously watches the API Server for Pod assignments and ensures containers are running correctly on the node.

Without kubelet:

Nodes cannot execute workloads
Pods cannot start
Cluster orchestration breaks completely

This is why kubelet is often called:

“The node agent of Kubernetes.”

What kubelet Actually Does

kubelet performs several critical operations continuously.

It:

Watches assigned Pods
Pulls container images
Starts containers
Monitors container health
Reports node status
Reports Pod status
Restarts failed containers
Executes liveness/readiness probes

kubelet is essentially the local orchestration engine running on every worker node.

kubelet Workflow (Step-by-Step)

Let’s understand how kubelet operates internally.

Step 1 — API Server Stores Desired State

Suppose you deploy:

replicas: 3

The API Server stores the Deployment state inside:

etcd

Step 2 — Scheduler Assigns Node

The Scheduler selects:

worker-node-2

The Pod object now contains:

spec.nodeName = worker-node-2

Step 3 — kubelet Detects Assignment

The kubelet running on:

worker-node-2

continuously watches the API Server.

It notices:

A Pod has been assigned to me

Step 4 — kubelet Pulls Container Image

kubelet communicates with the container runtime and requests:

Pull nginx image

The image may come from:

Docker Hub
Azure Container Registry
Amazon ECR
Private registries

Step 5 — Container Runtime Starts Container

The container runtime:

Creates namespaces
Allocates networking
Starts container process

The Pod now begins execution.

Step 6 — kubelet Reports Status

kubelet continuously reports back to the API Server:

Pod Running
Node Healthy
Container Ready

This status becomes visible through:

kubectl get pods

kubelet Continuously Monitors Workloads

One of the most important things to understand is:

kubelet never stops monitoring containers.

It continuously checks:

Container health
Probe results
Runtime status
Resource usage

If a container crashes:

kubelet attempts restart automatically

This contributes to Kubernetes self-healing behavior.

kubelet and Health Probes

kubelet executes:

Liveness probes
Readiness probes
Startup probes

These probes determine:

Whether containers are healthy
Whether traffic should be routed
Whether containers should restart

Example liveness probe:

livenessProbe:
  httpGet:
    path: /
    port: 80

If probe fails repeatedly:

kubelet restarts the container

Container Runtime — The Execution Engine

The container runtime is the software responsible for actually running containers.

Popular runtimes include:

Runtime	Description
containerd	Most common modern runtime
CRI-O	Kubernetes-focused runtime

Older Kubernetes versions used Docker directly, but modern Kubernetes primarily uses:

containerd

What Container Runtime Does

The runtime:

Pulls images
Creates containers
Starts processes
Manages namespaces
Handles cgroups
Controls isolation

It performs the low-level container operations that kubelet requests.

kube-proxy — The Networking Engine Behind Kubernetes Services

One of the most important challenges in Kubernetes networking is that Pods are temporary and constantly changing.

Whenever a Pod:

Crashes
Restarts
Gets rescheduled
Scales up or down

its IP address may change.

This creates a major networking problem because applications cannot reliably communicate using constantly changing Pod IPs.

Kubernetes solves this problem using:

Services

However, Services themselves are only logical Kubernetes objects. They do not actually forward traffic on their own.

The component responsible for making Kubernetes Services work is:

kube-proxy

kube-proxy is a networking component that runs on every worker node and manages how traffic reaches Pods inside the cluster.

Without kube-proxy:

Services cannot forward traffic
Internal load balancing breaks
Pod communication becomes unreliable
Kubernetes networking stops functioning properly

This is why kube-proxy is often considered:

“The networking engine of Kubernetes.”

What kube-proxy Actually Does

kube-proxy continuously watches the Kubernetes API Server for changes related to:

Services
Endpoints
Pods

kube-proxy is responsible for handling internal Kubernetes networking. It manages service routing, forwards traffic to healthy Pods, and dynamically updates load-balancing rules whenever Pods change inside the cluster.

Whenever something changes, kube-proxy dynamically updates networking rules on the node.

Its responsibilities include:

Responsibility	Purpose
Service networking	Enable Service communication
Traffic forwarding	Route requests to Pods
Load balancing	Distribute traffic
Endpoint tracking	Monitor healthy Pods
Networking rules	Configure iptables/IPVS

kube-proxy ensures traffic always reaches healthy backend Pods even when Pods constantly change.

Why Kubernetes Needs kube-proxy

Imagine a Deployment with:

3 nginx Pods

Each Pod gets its own IP address:

10.244.1.5
10.244.1.6
10.244.2.3

Now imagine:

One Pod crashes
Another Pod gets recreated
IP addresses change

Applications using direct Pod IPs would fail constantly.

Instead, Kubernetes creates a stable Service:

web-service

Applications communicate with:

web-service

instead of individual Pods.

kube-proxy makes this abstraction work behind the scenes.

How kube-proxy Works Internally

Suppose a request arrives at:

web-service

Here’s what happens internally.

Step 1 — Service Created

Suppose you create a Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80

This creates:

Stable virtual IP
Stable DNS name

However:

No actual process listens on this IP directly

Step 2 — kube-proxy Watches API Server

kube-proxy continuously watches:

Services
Endpoints
Pod changes

through the API Server.

It notices:

web-service has 3 backend Pods

Step 3 — kube-proxy Creates Routing Rules

kube-proxy dynamically creates networking rules using:

iptables
OR
IPVS

These rules map:

Service IP → Pod IPs

Example:

web-service → pod-1
web-service → pod-2
web-service → pod-3

Step 4 — Client Sends Request

Suppose an application sends:

curl http://web-service

The request reaches:

Service virtual IP

Step 5 — kube-proxy Routes Traffic

kube-proxy forwards the request to one of the healthy Pods automatically.

This creates:

Internal load balancing
Dynamic traffic routing
Stable networking abstraction

Request Flow in Kubernetes Networking

The traffic flow generally looks like this:

Client
   ↓
Kubernetes Service
   ↓
kube-proxy
   ↓
iptables/IPVS Rules
   ↓
Backend Pod

This entire process happens automatically.

Applications never need to know:

Which Pod IP changed
Which Pod restarted
Which node hosts the Pod

kube-proxy handles everything dynamically.

iptables vs IPVS

kube-proxy primarily works in two networking modes.

iptables Mode

iptables mode:

Uses Linux packet filtering rules
Very common in Kubernetes clusters
Simpler architecture

Traffic routing occurs through:

NAT rules
Packet forwarding chains

This mode works well for:

Small and medium clusters

IPVS Mode

IPVS mode is more advanced and scalable.

Advantages include:

Faster routing
Better load balancing
Improved performance
Better scalability

Large enterprise clusters usually prefer:

IPVS

because it handles high traffic volumes more efficiently.

Dynamic Endpoint Updates

One of the most powerful features of kube-proxy is:

Dynamic endpoint management

Suppose:

Pod crashes
New Pod gets created

kube-proxy automatically:

Removes failed Pod from routing
Adds replacement Pod
Updates networking rules instantly

Applications continue working without interruption.

This is essential for:

High availability
Auto scaling
Self-healing systems

Practical Lab — Create Deployment & Service

Now let’s see kube-proxy behavior practically.

Step 1 — Create Deployment

Create file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-demo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx

Apply:

kubectl apply -f deployment.yaml

Step 2 — Create Service

Create file:

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80

Apply:

kubectl apply -f service.yaml

Step 3 — Inspect Services

Run:

kubectl get svc

Example output:

NAME           TYPE        CLUSTER-IP
web-service    ClusterIP   10.96.15.10

This ClusterIP is managed internally through kube-proxy routing rules.

Step 4 — View Service Endpoints

Run:

kubectl get endpoints

Example:

web-service   10.244.1.5:80,10.244.2.7:80

These are the backend Pod IPs receiving traffic.

Step 5 — Describe Service

Run:

kubectl describe svc web-service

This displays:

ClusterIP
Ports
Endpoints
Selectors
Traffic mappings

🔬 Observe kube-proxy Pods

Most Kubernetes clusters run kube-proxy as a:

DaemonSet

This means:

One kube-proxy Pod runs per worker node

View kube-proxy Pods:

kubectl get pods -n kube-system

Example:

kube-proxy-abc123
kube-proxy-def456

Each node manages its own networking rules independently.

What Happens If kube-proxy Fails?

If kube-proxy stops functioning:

Services stop routing traffic
Internal communication breaks
Applications become unreachable
Load balancing fails

Pods may still run, but:

Service networking becomes unstable

Troubleshooting kube-proxy Issues

Common symptoms include:

Problem	Possible Cause
Service unreachable	kube-proxy failure
DNS resolves but traffic fails	Routing issue
Intermittent connectivity	Endpoint sync issue
Uneven traffic distribution	IPVS/iptables issue

Useful troubleshooting commands:

kubectl logs -n kube-system <kube-proxy-pod>

Enterprise Best Practices

Production Kubernetes clusters should:

Use IPVS for large-scale environments
Monitor kube-proxy health
Monitor Service latency
Avoid hardcoding Pod IPs
Use readiness probes correctly
Monitor endpoint changes carefully

Common Beginner Mistakes

Thinking Services are actual containers
Using direct Pod IPs in applications
Ignoring endpoint health
Misunderstanding ClusterIP networking
Assuming Services load balance equally always

Container Runtime Explained

The container runtime is the component responsible for actually running containers inside Kubernetes.

While Kubernetes is excellent at:

Scheduling workloads
Managing desired state
Orchestrating infrastructure

Kubernetes itself does not directly start containers.

Instead, Kubernetes delegates container execution to:

The Container Runtime

This runtime is installed on every worker node and works closely with:

kubelet
Linux kernel
Container images

Without a container runtime:

Pods cannot start
Containers cannot execute
Kubernetes workloads cannot run

This makes the container runtime one of the most critical components inside every Kubernetes worker node.

What Is a Container Runtime?

A container runtime is software responsible for:

Pulling container images
Creating containers
Starting container processes
Managing namespaces
Handling isolation
Managing container lifecycle

In simple terms:

The runtime is the engine that actually launches containers.

Kubernetes only decides:

WHAT should run
WHERE it should run

The runtime handles:

HOW the container actually executes

How Kubernetes Uses the Runtime

Suppose you deploy:

containers:
- name: nginx
  image: nginx

Internally:

kubelet receives the Pod assignment
kubelet contacts the container runtime
runtime pulls the image
runtime creates the container
runtime starts the process

The runtime performs the actual execution work.

Container Runtime Workflow

The internal flow usually looks like this:

API Server
     ↓
Scheduler
     ↓
kubelet
     ↓
Container Runtime
     ↓
Linux Kernel
     ↓
Container Process

Each component has a specialized role.

Popular Kubernetes Container Runtimes

Modern Kubernetes commonly uses:

Runtime	Description
containerd	Most common modern runtime
CRI-O	Kubernetes-native runtime

Older Kubernetes versions used Docker directly, but this changed significantly in recent Kubernetes releases.

containerd — Most Common Runtime

containerd is currently the most widely used Kubernetes runtime.

It is:

Lightweight
Stable
CNCF-supported
Optimized for Kubernetes

containerd handles:

Image pulling
Container lifecycle
Snapshot management
Runtime execution

Most managed Kubernetes services use:

containerd

including:

CRI-O — Kubernetes-Focused Runtime

CRI-O is another Kubernetes-focused runtime.

It was designed specifically for:

Kubernetes CRI compatibility

Advantages include:

Lightweight architecture
Kubernetes-native design
Reduced overhead

CRI-O is commonly used in:

OpenShift
Enterprise Red Hat environments

Docker vs Container Runtime

One of the biggest misconceptions in Kubernetes is:

“Kubernetes uses Docker.”

This was true historically, but modern Kubernetes architecture changed significantly.

Older Kubernetes Versions

Older Kubernetes versions used:

Docker Engine directly

The communication flow looked like this:

kubelet → Docker Engine → Containers

However, Docker was originally designed as:

Developer tooling
Container packaging platform

not specifically as a Kubernetes runtime.

Why Kubernetes Moved Away from Docker

Docker included many features Kubernetes did not actually need.

Example Docker components:

Image building
CLI tooling
Developer workflows
Registry integrations

Kubernetes mainly needed:

Container execution

This created unnecessary complexity.

The Docker Shim Problem

Kubernetes originally used a component called:

dockershim

dockershim acted as a translation layer between:

kubelet
Docker Engine

Architecture looked like:

kubelet
   ↓
dockershim
   ↓
Docker Engine
   ↓
Containers

This extra layer introduced:

Complexity
Maintenance overhead
Performance inefficiencies

Why Kubernetes Removed dockershim

Kubernetes eventually removed dockershim to improve:

Improvement	Benefit
Standardization	Unified runtime model
Stability	Fewer moving parts
Performance	Reduced overhead
Runtime flexibility	Support multiple runtimes

This transition pushed Kubernetes toward:

CRI-based runtimes

CRI — Container Runtime Interface

Kubernetes introduced:

CRI (Container Runtime Interface)

CRI is a standardized API that allows kubelet to communicate with any compatible runtime.

Now kubelet can interact with:

containerd
CRI-O
Other runtimes

through a consistent interface.

Architecture now looks like:

kubelet
   ↓
CRI
   ↓
containerd / CRI-O
   ↓
Containers

This architecture is:

Cleaner
More modular
Easier to maintain

What the Runtime Actually Does Internally

When kubelet asks the runtime to start a container, several low-level Linux operations occur.

The runtime:

Pulls image layers
Creates namespaces
Configures cgroups
Mounts filesystems
Allocates networking
Starts container process

This relies heavily on:

Linux kernel features

Linux Namespaces

Namespaces isolate:

Processes
Networking
Filesystems
Hostnames

This creates container separation.

cgroups

cgroups control:

CPU limits
Memory limits
Resource isolation

Example:

resources:
  limits:
    memory: "512Mi"

Container runtime enforces these limits through:

cgroups

Practical Lab — Inspect Container Runtime

Now let’s inspect the runtime directly.

Step 1 — View Node Details

Run:

kubectl describe node <node-name>

Look for:

Container Runtime Version

Example:

containerd://1.7.0

This shows:

Runtime type
Runtime version

Step 2 — View Node Information

Run:

kubectl get nodes -o wide

This displays:

Internal IPs
OS version
Runtime details

Step 3 — Inspect Running Pods

Run:

kubectl get pods -o wide

Observe:

Which Pods run on which nodes

The runtime on those nodes is executing the containers.

Optional Advanced Runtime Inspection

If you have node SSH access:

Check containerd:

systemctl status containerd

View running containers:

crictl ps

This directly interacts with the Kubernetes runtime layer.

What Happens If Runtime Fails?

If the container runtime crashes:

Containers may stop
New Pods cannot start
kubelet cannot manage workloads
Node may become unstable

Even if the Control Plane is healthy:

Workloads cannot execute without runtime support

Relationship Between kubelet and Runtime

A very important concept:

Component	Responsibility
kubelet	Node orchestration
Runtime	Container execution

kubelet:

Decides what should run locally

Runtime:

Actually launches containers

They work together continuously.

Complete Kubernetes Request Flow

Now let’s combine everything together.

Suppose you deploy:

kubectl apply -f deployment.yaml

Internally, Kubernetes performs the following sequence:

Step 1 — API Server Receives Request

kubectl sends YAML to:

kube-apiserver

Step 2 — Object Stored in etcd

Desired state gets saved inside:

etcd

Step 3 — Deployment Controller Detects Change

Controller Manager notices:

New Deployment created

Step 4 — ReplicaSet Created

Deployment Controller creates:

ReplicaSet

Step 5 — Pods Created

ReplicaSet creates:

Pod definitions

Step 6 — Scheduler Selects Node

Scheduler chooses:

worker-node-2

Step 7 — kubelet Receives Assignment

kubelet on worker-node-2 notices:

Pod assigned to me

Step 8 — Runtime Starts Containers

containerd or CRI-O:

Pulls image
Creates container
Starts process

Step 9 — kube-proxy Updates Routing

kube-proxy updates:

Service routing rules
Endpoint mappings

Step 10 — Pod Becomes Ready

Application becomes available.

This is:

Kubernetes architecture working together as a distributed orchestration system.

Why Architecture Knowledge Matters

Many engineers memorize Kubernetes commands but struggle during production incidents because they do not understand:

Internal component interaction
Control Plane behavior
Scheduling flow
Runtime execution
Networking architecture

Once you understand architecture:

Troubleshooting becomes logical
Failures become easier to diagnose
Cluster behavior becomes predictable
Enterprise operations become manageable

Common Kubernetes Architecture Mistakes

Many production problems occur because teams misunderstand Kubernetes internals.

Common mistakes include:

Ignoring etcd backups
Running single-node Control Plane in production
Monitoring Pods but ignoring Control Plane health
Misunderstanding Scheduler constraints
Assuming kubelet repairs failed nodes automatically
Hardcoding Pod IPs
Ignoring runtime failures

Enterprise Best Practices

Production Kubernetes environments should:

Monitor API Server latency
Backup etcd regularly
Use highly available Control Planes
Separate Control Plane nodes
Monitor Scheduler performance
Secure kubelet APIs
Monitor container runtime health
Monitor kube-proxy networking latency
Use proper resource requests and limits
Audit API requests

FAQs

What is the most important Kubernetes component?

The API Server is the central communication hub.

Why is etcd important?

etcd stores all cluster state and desired configuration.

What happens if Scheduler fails?

New Pods cannot be assigned to nodes.

What does kubelet do?

kubelet ensures containers run correctly on worker nodes.

Is kube-proxy required?

Yes, Services depend on kube-proxy for traffic routing.

Final Thoughts

Kubernetes architecture is the foundation of everything else in Kubernetes.

Once you deeply understand:

API Server
etcd
Scheduler
Controller Manager
kubelet
kube-proxy

Kubernetes stops feeling mysterious.

Instead, it becomes:

Predictable
Debuggable
Operable at scale

This knowledge is essential for:

CKA/CKS exams
Production troubleshooting
Platform engineering
Enterprise Kubernetes operations

GeekyMukesh

Week 3: Complete Kubernetes Architecture Guide

Control Plane, Worker Nodes & Internal Cluster Communication (Complete Practical Guide)

Who This Guide Is For

Why Kubernetes Architecture Matters

High-Level Kubernetes Architecture

The Control Plane (The Brain)

Kubernetes Architecture Overview

Core Control Plane Components

API Server — The Entry Point

What the API Server Does

Why Kubernetes Needs an API Server

Example Request Flow

Step 1 — kubectl Sends Request

Step 2 — Authentication

Step 3 — Authorization

Step 4 — Validation

Step 5 — Store State in etcd

Step 6 — Controllers Detect Changes

Step 7 — Scheduler Assigns Nodes

Step 8 — kubelet Starts Containers

Why Everything Uses the API Server

API Server Security Responsibilities

Admission Controllers

Kubernetes APIs

Practical Lab — Inspect API Server

Step 1: View Cluster Info

Step 2: View API Resources

View API Versions

Step 3: Access Health Endpoint

Step 4: View API Metrics

What Happens If API Server Fails?

etcd — The Cluster Database

What Does etcd Actually Store?

Suppose you run:

Step 1 — YAML Sent to API Server

Step 2 — API Server Validates Request

Step 3 — Object Stored in etcd

Step 4 — Controllers React

Desired State vs Actual State

Desired State

Actual State

Why etcd Is So Critical

What Happens If etcd Fails?

Possible Failures

Cluster State Loss

Scheduling Stops

API Server Becomes Unstable

Controllers Stop Reconciling

Why etcd Uses Distributed Consensus

etcd Architecture

Practical Lab — Explore etcd Objects

Step 1: Create ConfigMap

Step 2: Retrieve Object

Step 3: Understand State Persistence

Practical Lab — Observe Kubernetes State Storage

Step 1 — Create ConfigMap

Step 2 — Retrieve ConfigMap

Step 3 — Modify State

Practical Lab — Understand Desired State

Step 1 — Create Deployment

Step 2 — Delete a Pod

Step 3 — Observe Recovery

etcd Security

Enterprise Security Best Practices

Why etcd Backups Are Critical

Practical Lab — etcd Snapshot Backup

Why Managed Kubernetes Services Hide etcd

Common etcd Mistakes

Enterprise Best Practices

Final Thoughts

Scheduler — The Decision Maker

Scheduler Considers

Why Scheduling Is Important

What Happens When a Pod Is Created?

Step 1 — API Server Receives Request

Step 2 — Deployment Stored in etcd

Step 3 — Deployment Controller Creates Pods

Step 4 — Scheduler Detects Unscheduled Pods

Step 5 — Scheduler Evaluates Nodes