Pods

A Pod is an abstraction of a server
- It can run multiple containers within a single NameSpace, exposed by a single IP address
The Pod is the minimal entity that can be managed by Kubernetes
From a container perspective, a Pod is an entity that runs typically one or more containers by using container images
Typically, Pods are only started through a Deployment, because "naked" Pods are not rescheduled in case of a node failure

Naked Pods Disadvantage

Naked Pods are not rescheduled in case of failure
Rolling updates don't apply to naked Pods; you can only bring it down and bring it up again with the new settings
Naked Pods cannot be scaled
Naked pods cannot be replaced automatically

Managing Pods with `kubectl`

Use kubectl run to run a Pod based on a default image kubectl run myginx --image=nginx
kubectl get pods [-o yaml], the -o yaml option provides insight in all the Pod parameters.
kubectl describe pods shows all details about a Pod, including information about containers running within

Multi-Container Pods

The one-container Pod is the standard
Single container Pods are easier to build and maintain
Specific settings can be used that are good for a specific workload
To create applications that consist of multiple containers, microservices should be defined
In a microservice, different independently managed Pods are connected by resources that can be provided by Kubernetes

There are some defined cases where you might want to run multiple containers in a Pod

Sidecar container: a container that enhances the primary application, for instance for logging
Ambassador container: a container that represents the primary container to the outside world, such as a proxy
Adapter container: used to adopt the traffic or data pattern to match the traffic or data pattern in other applications in the cluster

Sidecar containers, etc. are not defined by specific Pod properties; from a Kubernetes API resource, it's just a multi-container Pod

Understanding the Sidecar Scenario

A sidecar container is providing additional functionality to the main container, where it makes no sense running this functionality in a separate Pod
Think of logging, monitoring, and syncing
The essence is that the main container and the sidecar container have access to shared resources to exchange information

Pod Condition

A pod has a pod status, which has an array of pod conditions through which the poh has or has not passed. Using kubectl describe pod PODNAME you can get the condition of pod.

These are the possible types -

PodSchedule : The pod has been scheduled to node
Ready : The pod is able to serve requests and will be added to the loaded balancing pools of all matching services
Initialized : All init containers have started successfully
Unscheduled : The scheduler cannot schedule the pod right now due to some reason it can be lack of resource or other constraints
ContainerReady : All containers in the ports are ready

Understanding Init Containers

An init container is an additional container in a Pod that completes a task before the "regular" container is started
The regular container will only be started once the init container has been started
If the init container has not run to completion, the main container is not started
Init Containers are specialized containers that run before app containers in a pod.
Init containers always run the completion
If a pod's init containers fail, kubernetes repeatedly restart the pod until the init container succeeds
Init containers do not support readiness probe

Use cases

Seeding a Database
Clone a git repository into a volume
Delaying the application launch until the dependencies are ready
Generate configuration file dynamically

For example (opens in a new tab) -

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app.kubernetes.io/name: MyApp
spec:
  containers:
    - name: myapp-container
      image: busybox:1.28
      command: ["sh", "-c", "echo The app is running! && sleep 3600"]
  initContainers:
    - name: init-myservice
      image: busybox:1.28
      command:
        [
          "sh",
          "-c",
          "until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done",
        ]
    - name: init-mydb
      image: busybox:1.28
      command:
        [
          "sh",
          "-c",
          "until nslookup mydb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mydb; sleep 2; done",
        ]

In the above example myapp-container won't start creating until the initContainer finish the job that is creating and running the container defined in InitContainer

Let's take a simple example of -

init-example.yaml

apiVersion: v1
kind: Pod
metadata:
  name: init-demo
spec:
  containers:
    - name: mynginx
      image: nginx
  initContainers:
    - name: mybusybox
      image: busybox
      command:
        - sleep
        - "3600"

As stated in example initContainer will create a container and run a sleep command for 3600 second to finish the job then nginx container creation will start

Output

controlplane $ kubectl get pods
NAME        READY   STATUS     RESTARTS   AGE
init-demo   0/1     Init:0/1   0          28s

Managing Pods with Advance features

Exploring Pod configuration

When deploying a Pod, many parameters are set to a default value
Use kubectl describe pod podname-xxx to see all these parameters and their current setting as they currently are in the etcd
Use documentation (opens in a new tab) for more information about these settings
Use kubectl get pods POD_NAME -o yaml to get the running pod detail into YAML format
In order to get more information about the specific properties from YAML file such as enableServiceLinks which is present in pods.specs.enableServiceLinks.

For that you can use kubectl explain pods.specs.enableServiceLinks

Connecting to a Pod for Furthur inspection

Apart from exploring a Pod externally, you can also connect to it and run commands on the primary container in a Pod:

kubectl exec -it nginx-xxx -- sh

Use sh shell because not every conatiner image use bash
- In this case it's not possibe to run all commands such as ps command.
- It's not found so we can use /proc file system that nginx is running. To find more info use cat 1/cmdline.
From here, run any command to investigate
Use exit to disconnect

Using the Pod logs

The Pod entrypoint application does not connect to any STDOUT
Application output is sent to the Kubernetes cluster
Use kubectl logs to connect to this output
Using kubectl logs is helpful in troubleshooting

Troubleshooting of failing application

Start by using kubectl get pods and check the status of your application
While seeing anything odd, use kubectl describe pod ... to get more information
If the Pod main application generates a non-zero exit code, use kubectl logs to figure out what is going wrong

Demo

Run a normal database pod

kubectl run mydb --image=mariadb

The pod will be created but if you look for the pod details you'll find that the STATUS is (ContainerCreating | Error | CrashLoopBackOff) but not Running.

Output

controlplane $ kubectl get pods
NAME   READY   STATUS   RESTARTS     AGE
mydb   0/1     Error    1 (7s ago)   21s

If that happen then we need to investgate and find the root cause and for that let's get the pod description in details.

Output

controlplane $ kubectl describe pods
Name:             mydb
Namespace:        default
.
.
.
Containers:
  mydb:
    Container ID:   containerd://e5909393bef08d519edaae64bde66eec5302c5aef32ad59ee1b29f0a123b1b88
    Image:          mariadb
    Image ID:       docker.io/library/mariadb@sha256:4a1de8fa2a929944373d7421105500ff6f889ce90dcb883fbb2fdb070e4d427e
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 20 Oct 2024 14:39:52 +0000
      Finished:     Sun, 20 Oct 2024 14:39:53 +0000
    Ready:          False
    Restart Count:  2
    Environment:    <none>
    .
    .
    .
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  46s                default-scheduler  Successfully assigned default/mydb to node01
  Normal   Pulled     34s                kubelet            Successfully pulled image "mariadb" in 12.217s (12.217s including waiting). Image size: 122621189 bytes.
  Normal   Pulled     30s                kubelet            Successfully pulled image "mariadb" in 804ms (804ms including waiting). Image size: 122621189 bytes.
  Normal   Pulling    16s (x3 over 46s)  kubelet            Pulling image "mariadb"
  Normal   Created    16s (x3 over 33s)  kubelet            Created container mydb
  Normal   Started    16s (x3 over 33s)  kubelet            Started container mydb
  Normal   Pulled     16s                kubelet            Successfully pulled image "mariadb" in 500ms (500ms including waiting). Image size: 122621189 bytes.
  Warning  BackOff    1s (x3 over 29s)   kubelet            Back-off restarting failed container mydb in pod mydb_default(3c94a589-3536-41b9-b8b7-dede0c349802)

We have the state is Waiting and the reason is CrashLoopBackOff i.e., the primary application doesn't start Successfully.
In kubernetes restartPolice is set true by default so it try again and again... .
Termination of pod reason is Error with EXit code 1 (Something is wrong from Application perspective). To check what's the error we need logs.

To get the logs for details of the Error

Output

controlplane $ kubectl logs mydb | tail -n 4
2024-10-20 15:16:39+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-10-20 15:16:39+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.5.2+maria~ubu2404 started.
2024-10-20 15:16:40+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
        You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ROOT_PASSWORD_HASH, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD

In order to run the database with environment variables -

Make sure to delete the pod because there is no way we can modify that/update that pod.

kubectl run mydb --image=mariadb --env MARIADB_ROOT_PASSWORD=password

Not a recommended way to pass the environment variables directly it should be passed using config and secrets

Verify the pod again if it's running

Output

controlplane $ kubectl logs mydb | tail -n 4
2024-10-20 15:24:29+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.5.2+maria~ubu2404 started.
2024-10-20 15:24:30+00:00 [Note] [Entrypoint]: Initializing database files

Using Port Forwarding to Access Pods

Pods can be accessed in multiple ways
A very simple way is by using port forwarding to expose a Pod port on the kubectl host that forwards to the Pod
kubectl port-forward fwnginx 8080:80
Port forwarding is useful for testing Pod accessibility on a specific cluster node, not to expose it to external users
Regular user access to applications is provided through services and ingress

Demo

Output

controlplane $ kubectl run myserver --image=httpd
pod/myserver created
controlplane $ kubectl get pods -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE              NOMINATED NODE   READINESS GATES
myserver   1/1     Running   0          7s    10.1.220.6   ip-172-31-24-47   <none>           <none>
controlplane $ curl 10.1.220.6
curl: (7) Failed to connect to 172.17.0.15 port 80: No route to host

As you can see we are not able to access the pod application that's because there is no port expose. To make it accessible we need something called port-forwarding which expose the port for a Pod.
This method is only for troubleshooting.

If you're using the tools like Minikube and MicroK8s port-forwarding might not be necessary. This is because by default, MicroK8s uses a single-node cluster where pods can access each other directly using their internal IP addresses within the cluster network.

Output

controlplane $ kubectl port-forwarding myserver 8080:80 &
controlplane $ curl 10.1.220.6
<h1>It works!</h1>

`& run the process in background. If it display the content in foreground press space it will go away.

Use fg command to bring the process in foreground

Understanding and Configuring SecurityContext

A SecurityContext defines privilege and access control settings for a Pod or container, and includes the following:

Discretionary Access Control which is about permissions used to access an object
Security Enhanced Linux, where security labels can be applied
Running as privileged or unprivileged user Using Linux capabilities
AppArmor, which is an alternative to SELinux
AllowPrivilegeEscalation which controls if a process can gain more privileges than its parent process

Use kubectl explain for a complete overview

Resource Limitation

By default, a Pod will use as much CPU and memory as necessary to do its work
This can be managed by using Memory/CPU requests and limits in pod.spec.containers.resources
A request is an initial request for resources
A limit defines the upper threshold of resources a Pod can use
Memory as well as CPU limits can be used
CPU limits are expressed in millicore or millicpu, 1/1000 of a CPU core, so, 500 millicore is 0.5 CPU
When being scheduled, the kube-scheduler ensures that the node running the Pods has all requested resources available
If a Pod with resource limits cannot be scheduled, it will show a status of Pending
When using a deployment, use kubectl set resources to change resource limitations on running applications
This doesn't work on Pods, as Pods don't provide an update mechanism
Pod/deployment resource limitations can be combined with quota on namespaces to restrict these applications in specific namespaces only

Demo

resource-limitation.yaml

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
    - name: db
      image: mysql
      env:
        - name: MYSQL_ROOT_PASSWORD
          value: "password"
      resources:
        requests:
          memory: "64Mi"
          cpu: "250m"
        limits:
          memory: "128Mi"
          cpu: "500m"
    - name: wp
      image: wordpress
      resources:
        requests:
          memory: "64Mi"
          cpu: "250m"
        limits:
          memory: "128Mi"
          cpu: "500m"

In the above YAML file we have configured the containers with less than what it requred/

kubectl get pods

Output

controlplane $ kubectl create -f file.yaml
pod/frontend created

kubectl describe pod frontend

Output

controlplane $ kubectl describe pods
Name:             frontend
Namespace:        default
.
.
.
Containers:
  db:
    Container ID:   containerd://5b149b4c52c69d6fd8ff423c1660980c4ea6f7d3ea01a260e6ee510a517773da
    Image:          mysql
    Image ID:       docker.io/library/mysql@sha256:2be51594eba5983f47e67ff5cb87d666a223e309c6c64450f30b5c59a788ea40
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 11 Nov 2024 21:23:25 +0000
      Finished:     Mon, 11 Nov 2024 21:23:32 +0000
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 11 Nov 2024 21:23:07 +0000
      Finished:     Mon, 11 Nov 2024 21:23:11 +0000
    Ready:          False
    Restart Count:  1
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      MYSQL_ROOT_PASSWORD:  password
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bq4xx (ro)
.
.
.
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  43s                default-scheduler  Successfully assigned default/frontend to node01
  Normal   Pulled     30s                kubelet            Successfully pulled image "mysql" in 12.735s (12.735s including waiting). Image size: 173882798 bytes.
  Normal   Pulling    30s                kubelet            Pulling image "wordpress"
  Normal   Pulled     14s                kubelet            Successfully pulled image "wordpress" in 16.203s (16.203s including waiting). Image size: 245514613 bytes.
  Normal   Pulling    13s (x2 over 43s)  kubelet            Pulling image "mysql"
  Normal   Created    13s                kubelet            Created container wp
  Normal   Started    13s                kubelet            Started container wp
  Normal   Created    12s (x2 over 30s)  kubelet            Created container db
  Normal   Started    12s (x2 over 30s)  kubelet            Started container db
  Normal   Pulled     12s                kubelet            Successfully pulled image "mysql" in 512ms (512ms including waiting). Image size: 173882798 bytes.
  Warning  BackOff    4s                 kubelet            Back-off restarting failed container db in pod frontend_default(a5a07125-3a5c-4c82-8104-847376a069fc)

OOMKilled is Out of Memory that is there is insufficient minimum memory to initiate the process to create a container.
Now let's configure the same YAML file with the configuration that is not possible to achieve like 256GB of RAM

resources:
  requests:
    memory: "256Gi"
    cpu: "250m"
  limits:
    memory: "512Gi"
    cpu: "500m"

Create the pod again and inspect the same.
If we provide the resource whoch is not possible or available physically then it will go in pending state.

Output

controlplane $ kubectl describe pods
Name:             frontend
Namespace:        default
.
.
.
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  14s   default-scheduler  0/2 nodes are available: 2 Insufficient memory. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.

To make it work fix he resources with

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "2Gi"
    cpu: "500m"

Understanding Quota

Quota are restrictions that are applied to namespaces
If Quota are set on a namespace, applications started in that namespace must have resource requests and limits set
Use kubectl create quota ... -n mynamespace to apply quota

Demo

Crating the quota (limitation) for a namespace

kubectl create namespace restricted
kubectl create quota mylimits -n restricted --hard=cpu=2,memory=1G,pods=3
kubectl describe ns restricted

Output

controlplane $ kubectl describe ns restricted
Name:         restricted
Labels:       kubernetes.io/metadata.name=restricted
Annotations:  <none>
Status:       Active

Resource Quotas
  Name:     mylimits
  Resource  Used  Hard
  --------  ---   ---
  cpu       0     2
  memory    0     1G
  pods      0     3

Running a pod in the namespace where we created quota. It will return an error because we can't scale the pods.

kubectl run pods restrictedpod --image=nginx -n restricted

Output

controlplane $ kubectl run pods restrictedpod --image=nginx -n restricted
Error from server (Forbidden): pods "pods" is forbidden: failed quota: mylimits: must specify cpu for: pods; memory for: pods

The best way to use quota is to create deployment with limits and request

Output

controlplane $ kubectl create deployment restricteddeploy -n restricted --image=nginx
deployment.apps/restricteddeploy created
controlplane $ kubectl set resources deploy restricteddeploy --limits=cpu=200m,memory=2G -n restricted
deployment.apps/restricteddeploy resource requirements updated
controlplane $ kubectl get all -n restricted
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/restricteddeploy   0/1     0            0           10s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/restricteddeploy-68d6898ddb   1         0         0       7s
replicaset.apps/restricteddeploy-8cf89bcbf    1         0         0       10s

We have set the quota but still not working, what's wrong?...Let's inspect it...

Output

controlplane $ kubectl describe deploy restricteddeploy -n restricted
Name:                   restricteddeploy
Namespace:              restricted
.
.
.
Conditions:
  Type             Status  Reason
  ----             ------  ------
  Progressing      True    NewReplicaSetCreated
  Available        False   MinimumReplicasUnavailable
  ReplicaFailure   True    FailedCreate
OldReplicaSets:    restricteddeploy-8cf89bcbf (0/1 replicas created)
NewReplicaSet:     restricteddeploy-68d6898ddb (0/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  2m29s  deployment-controller  Scaled up replica set restricteddeploy-8cf89bcbf to 1
  Normal  ScalingReplicaSet  2m26s  deployment-controller  Scaled up replica set restricteddeploy-68d6898ddb to 1

Inspecting the resplicaset for the root cause

Output

controlplane $ kubectl describe replicaset restricteddeploy-68d6898ddb -n restricted
Name:           restricteddeploy-68d6898ddb
Namespace:      restricted
.
.
.
Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate
Events:
  Type     Reason        Age                  From                   Message
  ----     ------        ----                 ----                   -------
  Warning  FailedCreate  3m51s                replicaset-controller  Error creating: pods "restricteddeploy-68d6898ddb-wd644" is forbidden: exceeded quota: mylimits, requested: memory=2G, used: memory=0, limited: memory=1G
.
.
.

It is found that the limit was set beyond the quota which was unacceptable. So we need to fix the limit and the request within the quota to make it work.

controlplane $ kubectl get all -n restricted
NAME                                   READY   STATUS    RESTARTS   AGE
pod/restricteddeploy-f46c84744-cgfgv   1/1     Running   0          10s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/restricteddeploy   1/1     1            1           4m55s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/restricteddeploy-68d6898ddb   0         0         0       4m52s
replicaset.apps/restricteddeploy-7f67cd48df   0         0         0       21s
replicaset.apps/restricteddeploy-8cf89bcbf    0         0         0       4m55s
replicaset.apps/restricteddeploy-f46c84744    1         1         1       10s

Cleaning up resources

Resources are not automatically cleaned up
Some resources have options for automatic cleanup if they're no longer used
Periodic manual cleanup may be required
If a Pod is managed by deployments, the deployment must be removed, not the Pod
Try not to force resource to deletion, it may bring them in an unmanageable state
If we don't cleanup the resource it will occupy the hardware unncessary. It may lead to multiple problems such as:
- Reduce Performance
- Increase bills
- Security
- Complex monitoring

Command	Description
`kubectl delete all <resource_name>`	Deletes a specific resource with the given name (e.g., Pod, Service) in the current namespace.
`kubectl delete all --all`	Deletes all resources (e.g., Pods, Services, Deployments, etc.) in the current namespace.
`kubectl delete all --all --force --grace-period=0`	Forcefully deletes all resources in the current namespace immediately without waiting for a grace period.

While kubectl delete all --all deletes all resources in the current namespace, it won't automatically recreate user-defined default resources. In some cases yes because the defaults are managed resources.

kubectl delete all --all -A --force --grace-period=0 - Don't do this in any circumstances

YAML in K8s Labels and Selectors

Pods

Naked Pods Disadvantage

Managing Pods with kubectl

Multi-Container Pods

Understanding the Sidecar Scenario

Pod Condition

Understanding Init Containers

Use cases

Managing Pods with Advance features

Exploring Pod configuration

Connecting to a Pod for Furthur inspection

Using the Pod logs

Troubleshooting of failing application

Using Port Forwarding to Access Pods

Understanding and Configuring SecurityContext

Resource Limitation

Understanding Quota

Cleaning up resources

Managing Pods with `kubectl`