Skip to Content
This project is a work in progress. If you have any questions or suggestions, feel free to contact me.
KubernetesWorkloadsPods

Pods

  • A Pod is an abstraction of a server
    • It can run multiple containers within a single NameSpace, exposed by a single IP address
  • The Pod is the minimal entity that can be managed by Kubernetes
  • From a container perspective, a Pod is an entity that runs typically one or more containers by using container images
  • Typically, Pods are only started through a Deployment, because “naked” Pods are not rescheduled in case of a node failure

Naked Pods Disadvantage

  • Naked Pods are not rescheduled in case of failure
  • Rolling updates don’t apply to naked Pods; you can only bring it down and bring it up again with the new settings
  • Naked Pods cannot be scaled
  • Naked pods cannot be replaced automatically

Managing Pods with kubectl

  • Use kubectl run to run a Pod based on a default image kubectl run myginx --image=nginx
  • kubectl get pods [-o yaml], the -o yaml option provides insight in all the Pod parameters.
  • kubectl describe pods shows all details about a Pod, including information about containers running within

Multi-Container Pods

  • The one-container Pod is the standard
  • Single container Pods are easier to build and maintain
  • Specific settings can be used that are good for a specific workload
  • To create applications that consist of multiple containers, microservices should be defined
  • In a microservice, different independently managed Pods are connected by resources that can be provided by Kubernetes

There are some defined cases where you might want to run multiple containers in a Pod

  • Sidecar container: a container that enhances the primary application, for instance for logging
  • Ambassador container: a container that represents the primary container to the outside world, such as a proxy
  • Adapter container: used to adopt the traffic or data pattern to match the traffic or data pattern in other applications in the cluster

Sidecar containers, etc. are not defined by specific Pod properties; from a Kubernetes API resource, it’s just a multi-container Pod

Understanding the Sidecar Scenario

  • A sidecar container is providing additional functionality to the main container, where it makes no sense running this functionality in a separate Pod
  • Think of logging, monitoring, and syncing
  • The essence is that the main container and the sidecar container have access to shared resources to exchange information

Pod Condition

A pod has a pod status, which has an array of pod conditions through which the poh has or has not passed. Using kubectl describe pod PODNAME you can get the condition of pod.

These are the possible types -

  • PodSchedule : The pod has been scheduled to node
  • Ready : The pod is able to serve requests and will be added to the loaded balancing pools of all matching services
  • Initialized : All init containers have started successfully
  • Unscheduled : The scheduler cannot schedule the pod right now due to some reason it can be lack of resource or other constraints
  • ContainerReady : All containers in the ports are ready

Understanding Init Containers

  • An init container is an additional container in a Pod that completes a task before the “regular” container is started
  • The regular container will only be started once the init container has been started
  • If the init container has not run to completion, the main container is not started
  • Init Containers are specialized containers that run before app containers in a pod.
  • Init containers always run the completion
  • If a pod’s init containers fail, kubernetes repeatedly restart the pod until the init container succeeds
  • Init containers do not support readiness probe

Use cases

  • Seeding a Database
  • Clone a git repository into a volume
  • Delaying the application launch until the dependencies are ready
  • Generate configuration file dynamically

For example -

apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app.kubernetes.io/name: MyApp spec: containers: - name: myapp-container image: busybox:1.28 command: ["sh", "-c", "echo The app is running! && sleep 3600"] initContainers: - name: init-myservice image: busybox:1.28 command: [ "sh", "-c", "until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done", ] - name: init-mydb image: busybox:1.28 command: [ "sh", "-c", "until nslookup mydb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mydb; sleep 2; done", ]

In the above example myapp-container won’t start creating until the initContainer finish the job that is creating and running the container defined in InitContainer

Let’s take a simple example of -

init-example.yaml
apiVersion: v1 kind: Pod metadata: name: init-demo spec: containers: - name: mynginx image: nginx initContainers: - name: mybusybox image: busybox command: - sleep - "3600"

As stated in example initContainer will create a container and run a sleep command for 3600 second to finish the job then nginx container creation will start

Output
controlplane $ kubectl get pods NAME READY STATUS RESTARTS AGE init-demo 0/1 Init:0/1 0 28s

Managing Pods with Advance features

Exploring Pod configuration

  • When deploying a Pod, many parameters are set to a default value
  • Use kubectl describe pod podname-xxx to see all these parameters and their current setting as they currently are in the etcd
  • Use documentation for more information about these settings
  • Use kubectl get pods POD_NAME -o yaml to get the running pod detail into YAML format
  • In order to get more information about the specific properties from YAML file such as enableServiceLinks which is present in pods.specs.enableServiceLinks.

For that you can use kubectl explain pods.specs.enableServiceLinks

Connecting to a Pod for Furthur inspection

Apart from exploring a Pod externally, you can also connect to it and run commands on the primary container in a Pod:

  • kubectl exec -it nginx-xxx -- sh

    Use sh shell because not every conatiner image use bash

    • In this case it’s not possibe to run all commands such as ps command.
    • It’s not found so we can use /proc file system that nginx is running. To find more info use cat 1/cmdline.
  • From here, run any command to investigate

  • Use exit to disconnect

Using the Pod logs

  • The Pod entrypoint application does not connect to any STDOUT
  • Application output is sent to the Kubernetes cluster
  • Use kubectl logs to connect to this output
  • Using kubectl logs is helpful in troubleshooting

Troubleshooting of failing application

  • Start by using kubectl get pods and check the status of your application
  • While seeing anything odd, use kubectl describe pod ... to get more information
  • If the Pod main application generates a non-zero exit code, use kubectl logs to figure out what is going wrong

Demo

Run a normal database pod

kubectl run mydb --image=mariadb

The pod will be created but if you look for the pod details you’ll find that the STATUS is (ContainerCreating | Error | CrashLoopBackOff) but not Running.

Output
controlplane $ kubectl get pods NAME READY STATUS RESTARTS AGE mydb 0/1 Error 1 (7s ago) 21s

If that happen then we need to investgate and find the root cause and for that let’s get the pod description in details.

Output
controlplane $ kubectl describe pods Name: mydb Namespace: default . . . Containers: mydb: Container ID: containerd://e5909393bef08d519edaae64bde66eec5302c5aef32ad59ee1b29f0a123b1b88 Image: mariadb Image ID: docker.io/library/mariadb@sha256:4a1de8fa2a929944373d7421105500ff6f889ce90dcb883fbb2fdb070e4d427e Port: <none> Host Port: <none> State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Sun, 20 Oct 2024 14:39:52 +0000 Finished: Sun, 20 Oct 2024 14:39:53 +0000 Ready: False Restart Count: 2 Environment: <none> . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 46s default-scheduler Successfully assigned default/mydb to node01 Normal Pulled 34s kubelet Successfully pulled image "mariadb" in 12.217s (12.217s including waiting). Image size: 122621189 bytes. Normal Pulled 30s kubelet Successfully pulled image "mariadb" in 804ms (804ms including waiting). Image size: 122621189 bytes. Normal Pulling 16s (x3 over 46s) kubelet Pulling image "mariadb" Normal Created 16s (x3 over 33s) kubelet Created container mydb Normal Started 16s (x3 over 33s) kubelet Started container mydb Normal Pulled 16s kubelet Successfully pulled image "mariadb" in 500ms (500ms including waiting). Image size: 122621189 bytes. Warning BackOff 1s (x3 over 29s) kubelet Back-off restarting failed container mydb in pod mydb_default(3c94a589-3536-41b9-b8b7-dede0c349802)
  • We have the state is Waiting and the reason is CrashLoopBackOff i.e., the primary application doesn’t start Successfully.
  • In kubernetes restartPolice is set true by default so it try again and again… .
  • Termination of pod reason is Error with EXit code 1 (Something is wrong from Application perspective). To check what’s the error we need logs.

To get the logs for details of the Error

Output
controlplane $ kubectl logs mydb | tail -n 4 2024-10-20 15:16:39+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' 2024-10-20 15:16:39+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.5.2+maria~ubu2404 started. 2024-10-20 15:16:40+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ROOT_PASSWORD_HASH, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD

In order to run the database with environment variables -

Make sure to delete the pod because there is no way we can modify that/update that pod.

kubectl run mydb --image=mariadb --env MARIADB_ROOT_PASSWORD=password

Not a recommended way to pass the environment variables directly it should be passed using config and secrets

Verify the pod again if it’s running

Output
controlplane $ kubectl logs mydb | tail -n 4 2024-10-20 15:24:29+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.5.2+maria~ubu2404 started. 2024-10-20 15:24:30+00:00 [Note] [Entrypoint]: Initializing database files

Using Port Forwarding to Access Pods

  • Pods can be accessed in multiple ways
  • A very simple way is by using port forwarding to expose a Pod port on the kubectl host that forwards to the Pod
  • kubectl port-forward fwnginx 8080:80
  • Port forwarding is useful for testing Pod accessibility on a specific cluster node, not to expose it to external users
  • Regular user access to applications is provided through services and ingress

Demo

Output
controlplane $ kubectl run myserver --image=httpd pod/myserver created controlplane $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myserver 1/1 Running 0 7s 10.1.220.6 ip-172-31-24-47 <none> <none> controlplane $ curl 10.1.220.6 curl: (7) Failed to connect to 172.17.0.15 port 80: No route to host
  • As you can see we are not able to access the pod application that’s because there is no port expose. To make it accessible we need something called port-forwarding which expose the port for a Pod.
  • This method is only for troubleshooting.

If you’re using the tools like Minikube and MicroK8s port-forwarding might not be necessary. This is because by default, MicroK8s uses a single-node cluster where pods can access each other directly using their internal IP addresses within the cluster network.

Output
controlplane $ kubectl port-forwarding myserver 8080:80 & controlplane $ curl 10.1.220.6 <h1>It works!</h1>
  • `& run the process in background. If it display the content in foreground press space it will go away.
  • Use fg command to bring the process in foreground

Understanding and Configuring SecurityContext

A SecurityContext defines privilege and access control settings for a Pod or container, and includes the following:

  • Discretionary Access Control which is about permissions used to access an object
  • Security Enhanced Linux, where security labels can be applied
  • Running as privileged or unprivileged user Using Linux capabilities
  • AppArmor, which is an alternative to SELinux
  • AllowPrivilegeEscalation which controls if a process can gain more privileges than its parent process

Use kubectl explain for a complete overview

Resource Limitation

  • By default, a Pod will use as much CPU and memory as necessary to do its work
  • This can be managed by using Memory/CPU requests and limits in pod.spec.containers.resources
  • A request is an initial request for resources
  • A limit defines the upper threshold of resources a Pod can use
  • Memory as well as CPU limits can be used
  • CPU limits are expressed in millicore or millicpu, 1/1000 of a CPU core, so, 500 millicore is 0.5 CPU
  • When being scheduled, the kube-scheduler ensures that the node running the Pods has all requested resources available
  • If a Pod with resource limits cannot be scheduled, it will show a status of Pending
  • When using a deployment, use kubectl set resources to change resource limitations on running applications
  • This doesn’t work on Pods, as Pods don’t provide an update mechanism
  • Pod/deployment resource limitations can be combined with quota on namespaces to restrict these applications in specific namespaces only

Demo

resource-limitation.yaml
apiVersion: v1 kind: Pod metadata: name: frontend spec: containers: - name: db image: mysql env: - name: MYSQL_ROOT_PASSWORD value: "password" resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m" - name: wp image: wordpress resources: requests: memory: "64Mi" cpu: "250m" limits: memory: "128Mi" cpu: "500m"
  • In the above YAML file we have configured the containers with less than what it requred/
kubectl get pods
Output
controlplane $ kubectl create -f file.yaml pod/frontend created
kubectl describe pod frontend
Output
controlplane $ kubectl describe pods Name: frontend Namespace: default . . . Containers: db: Container ID: containerd://5b149b4c52c69d6fd8ff423c1660980c4ea6f7d3ea01a260e6ee510a517773da Image: mysql Image ID: docker.io/library/mysql@sha256:2be51594eba5983f47e67ff5cb87d666a223e309c6c64450f30b5c59a788ea40 Port: <none> Host Port: <none> State: Terminated Reason: OOMKilled Exit Code: 137 Started: Mon, 11 Nov 2024 21:23:25 +0000 Finished: Mon, 11 Nov 2024 21:23:32 +0000 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Mon, 11 Nov 2024 21:23:07 +0000 Finished: Mon, 11 Nov 2024 21:23:11 +0000 Ready: False Restart Count: 1 Limits: cpu: 500m memory: 128Mi Requests: cpu: 250m memory: 64Mi Environment: MYSQL_ROOT_PASSWORD: password Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bq4xx (ro) . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43s default-scheduler Successfully assigned default/frontend to node01 Normal Pulled 30s kubelet Successfully pulled image "mysql" in 12.735s (12.735s including waiting). Image size: 173882798 bytes. Normal Pulling 30s kubelet Pulling image "wordpress" Normal Pulled 14s kubelet Successfully pulled image "wordpress" in 16.203s (16.203s including waiting). Image size: 245514613 bytes. Normal Pulling 13s (x2 over 43s) kubelet Pulling image "mysql" Normal Created 13s kubelet Created container wp Normal Started 13s kubelet Started container wp Normal Created 12s (x2 over 30s) kubelet Created container db Normal Started 12s (x2 over 30s) kubelet Started container db Normal Pulled 12s kubelet Successfully pulled image "mysql" in 512ms (512ms including waiting). Image size: 173882798 bytes. Warning BackOff 4s kubelet Back-off restarting failed container db in pod frontend_default(a5a07125-3a5c-4c82-8104-847376a069fc)
  • OOMKilled is Out of Memory that is there is insufficient minimum memory to initiate the process to create a container.
  • Now let’s configure the same YAML file with the configuration that is not possible to achieve like 256GB of RAM
resources: requests: memory: "256Gi" cpu: "250m" limits: memory: "512Gi" cpu: "500m"
  • Create the pod again and inspect the same.
  • If we provide the resource whoch is not possible or available physically then it will go in pending state.
Output
controlplane $ kubectl describe pods Name: frontend Namespace: default . . . Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 14s default-scheduler 0/2 nodes are available: 2 Insufficient memory. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod.
  • To make it work fix he resources with
resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "2Gi" cpu: "500m"

Understanding Quota

  • Quota are restrictions that are applied to namespaces
  • If Quota are set on a namespace, applications started in that namespace must have resource requests and limits set
  • Use kubectl create quota ... -n mynamespace to apply quota

Demo

  • Crating the quota (limitation) for a namespace
kubectl create namespace restricted kubectl create quota mylimits -n restricted --hard=cpu=2,memory=1G,pods=3 kubectl describe ns restricted
Output
controlplane $ kubectl describe ns restricted Name: restricted Labels: kubernetes.io/metadata.name=restricted Annotations: <none> Status: Active Resource Quotas Name: mylimits Resource Used Hard -------- --- --- cpu 0 2 memory 0 1G pods 0 3
  • Running a pod in the namespace where we created quota. It will return an error because we can’t scale the pods.
kubectl run pods restrictedpod --image=nginx -n restricted
Output
controlplane $ kubectl run pods restrictedpod --image=nginx -n restricted Error from server (Forbidden): pods "pods" is forbidden: failed quota: mylimits: must specify cpu for: pods; memory for: pods
  • The best way to use quota is to create deployment with limits and request
Output
controlplane $ kubectl create deployment restricteddeploy -n restricted --image=nginx deployment.apps/restricteddeploy created controlplane $ kubectl set resources deploy restricteddeploy --limits=cpu=200m,memory=2G -n restricted deployment.apps/restricteddeploy resource requirements updated controlplane $ kubectl get all -n restricted NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/restricteddeploy 0/1 0 0 10s NAME DESIRED CURRENT READY AGE replicaset.apps/restricteddeploy-68d6898ddb 1 0 0 7s replicaset.apps/restricteddeploy-8cf89bcbf 1 0 0 10s

We have set the quota but still not working, what’s wrong?…Let’s inspect it…

Output
controlplane $ kubectl describe deploy restricteddeploy -n restricted Name: restricteddeploy Namespace: restricted . . . Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetCreated Available False MinimumReplicasUnavailable ReplicaFailure True FailedCreate OldReplicaSets: restricteddeploy-8cf89bcbf (0/1 replicas created) NewReplicaSet: restricteddeploy-68d6898ddb (0/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 2m29s deployment-controller Scaled up replica set restricteddeploy-8cf89bcbf to 1 Normal ScalingReplicaSet 2m26s deployment-controller Scaled up replica set restricteddeploy-68d6898ddb to 1
  • Inspecting the resplicaset for the root cause
Output
controlplane $ kubectl describe replicaset restricteddeploy-68d6898ddb -n restricted Name: restricteddeploy-68d6898ddb Namespace: restricted . . . Conditions: Type Status Reason ---- ------ ------ ReplicaFailure True FailedCreate Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 3m51s replicaset-controller Error creating: pods "restricteddeploy-68d6898ddb-wd644" is forbidden: exceeded quota: mylimits, requested: memory=2G, used: memory=0, limited: memory=1G . . .

It is found that the limit was set beyond the quota which was unacceptable. So we need to fix the limit and the request within the quota to make it work.

controlplane $ kubectl get all -n restricted NAME READY STATUS RESTARTS AGE pod/restricteddeploy-f46c84744-cgfgv 1/1 Running 0 10s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/restricteddeploy 1/1 1 1 4m55s NAME DESIRED CURRENT READY AGE replicaset.apps/restricteddeploy-68d6898ddb 0 0 0 4m52s replicaset.apps/restricteddeploy-7f67cd48df 0 0 0 21s replicaset.apps/restricteddeploy-8cf89bcbf 0 0 0 4m55s replicaset.apps/restricteddeploy-f46c84744 1 1 1 10s

Cleaning up resources

  • Resources are not automatically cleaned up

  • Some resources have options for automatic cleanup if they’re no longer used

  • Periodic manual cleanup may be required

  • If a Pod is managed by deployments, the deployment must be removed, not the Pod

  • Try not to force resource to deletion, it may bring them in an unmanageable state

  • If we don’t cleanup the resource it will occupy the hardware unncessary. It may lead to multiple problems such as:

    • Reduce Performance
    • Increase bills
    • Security
    • Complex monitoring
CommandDescription
kubectl delete all <resource_name>Deletes a specific resource with the given name (e.g., Pod, Service) in the current namespace.
kubectl delete all --allDeletes all resources (e.g., Pods, Services, Deployments, etc.) in the current namespace.
kubectl delete all --all --force --grace-period=0Forcefully deletes all resources in the current namespace immediately without waiting for a grace period.
  • While kubectl delete all --all deletes all resources in the current namespace, it won’t automatically recreate user-defined default resources. In some cases yes because the defaults are managed resources.
  • kubectl delete all --all -A --force --grace-period=0 - Don’t do this in any circumstances
Last updated on