How Kubernetes manages storage resources.

First we will learn about Volume and how Kubernetes provides storage for containers in clusters through Volume; then we will practice several commonly used Volume types and understand their respective application scenarios; finally, we will discuss how Kubernetes separates cluster administrators from cluster users through Persistent Volume and Persistent Volume Claim, and practice Vo. Static and dynamic supply of lume.

Volume

In this section, we discuss Kubernetes'storage model Volume and learn how to map various persistent storage to containers.

We often say that containers and pods are short-lived.
The implication is that their life cycles may be short and they will be destroyed and created frequently. When the container is destroyed, the data stored in the file system inside the container will be cleared.

To persist the container's data, you can use Kubernetes Volume.

Volume's life cycle is independent of containers. Containers in Pod may be destroyed and rebuilt, but Volume will be retained.

In essence, Kubernetes Volume is a directory, similar to Docker Volume. When Volume is mount ed to Pod, all containers in Pod can access the Volume. Kubernetes Volume also supports a variety of backend types, including emptyDir, hostPath, GCE Persistent Disk, AWS Elastic Block Store, NFS, Ceph, etc. The complete list is available for reference. https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes

Volume provides an abstraction of various backend s. When using Volume to read and write data, the container does not need to care whether the data is stored in the file system of the local node or on the cloud hard disk. For it, all types of Volume are just a directory.

We'll start with the simplest emptyDir, Kubernetes Volume.

emptyDir

EmptyDir is the most basic Volume type. As its name indicates, an emptyDir Volume is an empty directory on the Host.

emptyDir Volume is persistent for containers, but not for Pods. When a Pod is deleted from a node, the Volume content is also deleted. But if only the container was destroyed and Pod was still there, Volume would not be affected.

That is to say, the life cycle of emptyDir Volume is the same as that of Pod.

All containers in Pod can share Volume, and they can specify their mount paths. Following is an example to practice emptyDir. The configuration file is as follows:

apiVersion: v1
kind: Pod
metadata:
  name: producer-consumer
spec:
  containers:
  - image: busybox
    name: producer
    volumeMounts:
    - mountPath: /producer_dir
      name: shared-volume
    args:
    - /bin/sh
    - -c
    - echo "hello world"> /producer_dir/hello ; sleep 30000

  - image: busybox
    name: consumer
    volumeMounts:
    - mountPath: /consumer_dir
      name: shared-volume
    args:
    - /bin/sh
    - -c
    - cat /consumer_dir/hello ; sleep 30000

  volumes:
  - name: shared-volume
    emptyDir: {}

Here we simulate a producer-consumer scenario. Pod has two containers, producer and consumer, which share a Volume. Producer is responsible for writing data to Volume, while consumer reads data from Volume.

The bottom volume of the file defines a Volume shared-volume of emptyDir type.

(2) The producer container will share-volume mount to the / producer_dir directory.

(3) The producer writes the data to the file hello through echo.

(4) The consumer container will share-volume mount to the / consumer_dir directory.

consumer reads data from the file hello through cat.

Execute the following command to create a Pod:

# kubectl apply -f emptyDir.yaml
pod/producer-consumer created

# kubectl get pod
NAME                READY   STATUS    RESTARTS   AGE
producer-consumer   2/2     Running   0          87s

# kubectl logs producer-consumer consumer 
hello world

kubectl logs show that the container consumer successfully reads the data written by producer, verifying that the two containers share emptyDir Volume.

Because emptyDir is a directory in the Docker Host filesystem, its effect is equivalent to executing docker run-v/producer_dir and docker run-v/consumer_dir. Looking at the container configuration details through docker inspect, we found that both containers mount ed the same directory:

 {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/188767a3-cf18-4bf9-a89b-b0c4cf4124bb/volumes/kubernetes.io~empty-dir/shared-volume",
                "Destination": "/consumer_dir",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }

"Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/kubelet/pods/188767a3-cf18-4bf9-a89b-b0c4cf4124bb/volumes/kubernetes.io~empty-dir/shared-volume",
                "Destination": "/producer_dir",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },

Here/var/lib/kubelet/pods/3e6100eb-a97a-11e7-8f72-0800274451ad/volumes/kubernetes.io~empty-dir/shared-volume is the real path of emptyDir on Host.

EmptyDir is a temporary directory created on Host. Its advantage is that it can easily provide shared storage for containers in Pod without additional configuration. But it doesn't have persistence, and if Pod doesn't exist, emptyDir doesn't exist. Based on this feature, emptyDir is particularly suited for situations where containers in Pod need temporary shared storage space, such as the previous producer-consumer use case.

hostPath Volume.

The function of hostPath Volume is to mount the existing directory in the Docker Host file system to the container of Pod. Most applications do not use hostPath Volume because it actually increases the coupling between Pod and nodes, limiting the use of Pod. However, applications that need access to Kubernetes or Docker internal data (configuration files and binary libraries) need to use hostPath.

For example, kube-apiserver and kube-controller-manager are such applications, through which

kubectl edit --namespace=kube-system pod kube-apiserver-k8s-master
To see the configuration of kube-apiserver Pod, the following is the relevant section of Volume:

   volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true

  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs

Three hostPath volume k8s, certs and PKI are defined, which correspond to the Host directory / etc/kubernetes, / etc/ssl/certs and / etc/pki, respectively.

If the Pod is destroyed, the directory corresponding to the hostpath will also be preserved. From this point of view, the persistence of hostpath is better than emptyDir. But once Host crashes, hostPath can't be accessed.

A truly persistent Volume.

External Storage Provider

If Kubernetes is deployed on public clouds such as AWS, GCE, Azure, you can directly use the cloud hard disk as Volume. Here is an example of AWS Elastic Block Store:

To use ESB volume in Pod, you must first create it in AWS and then refer to it through volume-id. The use of other cloud hard drives can be referred to the official documents of various public cloud vendors.

Kubernetes Volume can also use mainstream distributed storage, such as Ceph, GlusterFS, etc. the following is an example of Ceph:

Ceph's / some/path/in/side/cephfs directory is mount ed to the container path / test-ceph.

Compared with emptyDir and hostPath, the biggest feature of these Volume types is that they do not rely on Kubernetes. Volume's underlying infrastructure is managed by an independent storage system, separated from the Kubernetes cluster. After data is persisted, even the entire Kubernetes crash will not be compromised.

Of course, storage systems such as operations and maintenance are usually not a simple task, especially when there are high requirements for reliability, high availability and scalability.

Volume provides a very good data persistence solution, but there are still shortcomings in manageability. In the next section, we'll learn about storage solutions with more executive rationality: Persistent Volume & Persistent Volume Claim.

PV & PVC

Volume provides a very good data persistence solution, but there are still shortcomings in manageability.

Take the previous AWS EBS example. To use Volume, Pod must know the following information beforehand:

  • Currently Volume comes from AWS EBS.
  • EBS Volume has been created beforehand and knows the exact volume-id.

Pod is usually maintained by the application developer, while Volume is usually maintained by the administrator of the storage system. Developers need to get the above information:

  • Or ask the administrator.
  • Either they are administrators themselves.

This leads to a management problem: the responsibilities of application developers and system administrators are coupled. It is acceptable if the system is small or for a development environment. However, when the cluster size becomes larger, especially for the generating environment, considering efficiency and security, this has become a problem that must be solved.

Kubernetes's solution is Persistent Volume and Persistent Volume Claim.

Persistent volume (PV) is a piece of storage space in the external storage system, which is created and maintained by the administrator. Like Volume, PV is persistent and lifecycle independent of Pod.

Persistent volume claim (PVC) is a claim for PV. PVC is usually created and maintained by ordinary users. When storage resources need to be allocated to Pod, users can create a PVC that specifies the size of storage resources and access modes (such as read-only), and Kubernetes will find and provide PVs that meet the requirements.

With Persistent Volume Claim, users only need to tell Kubernetes what kind of storage resources they need, without having to care about where the real space is allocated and how to access the underlying details. The underlying information of the Storage Provider is handled by the administrator, who should only care about the details of creating Persistent Volume.

Kubernetes supports a variety of Persistent Volumes, such as AWS EBS, Ceph, NFS, etc. For a complete list, see https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes

NFS PersistentVolume

Practice PV and PVC through NFS.

As a preparation, we have built an NFS server on the k8s-master node with the directory / nfsdata:

[root@k8s-master ~]# showmount -e
Export list for k8s-master:
/nfsdata 192.168.168.0/24

Next, create a PV mypv1 configuration file nfs-pv1.yml as follows:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: nfs
  nfs:
    path: /nfsdata/pv1
    server: 192.168.168.10

(1) Capacity specifies that the capacity of PV is 1G.

(2) accessModes specifies the access mode as ReadWriteOnce, and supports the following access modes:
ReadWriteOnce-PV can mount to a single node in read-write mode.
ReadOnlyMany-PV can mount to multiple nodes in read-only mode.
ReadWriteMany-PV can mount to multiple nodes in read-write mode.

(3) The persistent Volume Reclaim Policy specifies that when the PV recovery strategy is Recycle, the supporting strategies are:
Retain - Manual recycling by administrators is required.
Recycle --- Clear the data in PV, the effect is equivalent to executing rm-rf/the volume/*.
Delete - Delete the corresponding storage resources on Storage Provider, such as AWS EBS, GCE PD, Azure Disk, OpenStack Cinder Volume, etc.

Storage ClassName specifies that the class of PV is nfs. It is equivalent to setting up a classification for PV. PVC can specify the PV of the corresponding class to apply for.

(5) Specify the corresponding directory of PV on the NFS server.

Create mypv1:

[root@k8s-master ~]# kubectl apply -f nfs-pv1.yml 
persistentvolume/mypv1 created
[root@k8s-master ~]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Recycle          Available           nfs                     7s

STATUS is Available, indicating that mypv1 is ready and can be applied for by PVC.

Next, create PVC mypvc1. The configuration file nfs-pvc1.yml is as follows:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mypvc1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs

PVC is very simple, just specify the PV capacity, access mode and class.

Create mypvc1:

[root@k8s-master ~]# kubectl apply -f nfs-pvc1.yml 
persistentvolumeclaim/mypvc1 created
[root@k8s-master ~]# kubectl get pvc
NAME     STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc1   Bound    mypv1    1Gi        RWO            nfs            6s
[root@k8s-master ~]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE
mypv1   1Gi        RWO            Recycle          Bound    default/mypvc1   nfs                     2m18s
[root@k8s-master ~]# 

From the output of kubectl get pvc and kubectl get pv, we can see that mypvc1 has Bound to mypv1, and the application is successful.

Next you can use storage in Pod. The Pod configuration file pod1.yml is as follows:

kind: Pod
apiVersion: v1
metadata:
  name: mypod1
spec:
  containers:
    - name: mypod1
      image: busybox
      args:
      - /bin/sh
      - -c
      - sleep 30000
      volumeMounts:
      - mountPath: "/mydata"
        name: mydata
  volumes:
    - name: mydata
      persistentVolumeClaim:
        claimName: mypvc1

Similar to the normal Volume format, in volumes, you specify the Volume to be applied for using mypvc1 through persistent Volume Claim.

Create mypod1:

[root@k8s-master ~]# kubectl apply -f pod1.yml
pod/mypod1 created

[root@k8s-master ~]# kubectl get pod -o wide     
NAME     READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
mypod1   1/1     Running   0          3m19s   10.244.1.2   k8s-node1   <none>           <none>

Verify that PV is available:

[root@k8s-master ~]# kubectl exec mypod1 touch /mydata/hello
[root@k8s-master ~]# ll /nfsdata/pv1/
total 0
-rw-r--r-- 1 root root 0 Oct 12 16:36 hello

As you can see, the file / mydata/hello created in Pod has indeed been saved in the NFS server directory / nfsdata/pv1.

If you no longer need to use PV, you can remove PVC and recycle PV.

How does MySQL use PV and PVC?

This section demonstrates how to provide persistent storage for MySQL databases by:

Create PV and PVC.

Deploy MySQL.

Add data to MySQL.

Kubernetes automatically migrates MySQL to other nodes to simulate node downtime.

Verify data consistency.

First, create PV and PVC with the following configuration:

mysql-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    path: /nfsdata/mysql-pv
    server: 192.168.77.10

mysql-pvc.yml

kind: PersistentVolumeClaim
apiVersion: v1
metadata: 
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs

Create mysql-pv and mysql-pvc:

Next, deploy MySQL with the following configuration files:

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pvc  

PVC mysql-pvc Bound PV mysql-pv will be mount ed to MySQL data directory var/lib/mysql.

MySQL is deployed to k8s-node 2, and the following Service mysql is accessed through the client:

# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword 

Update the database:

[root@k8s-master ~]# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
If you don't see a command prompt, try pressing enter.

mysql> use mysql 
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> create table my_id(id int(4));
Query OK, 0 rows affected (0.09 sec)

mysql> insert into my_id values(111);
Query OK, 1 row affected (0.00 sec)

mysql> select * from my_id;
+------+
| id   |
+------+
|  111 |
+------+
1 row in set (0.00 sec)

Switch to database mysql.

(2) Create the database table my_id.

(3) Insert a piece of data.

(4) Verify that the data has been written.

Turn k8s-node 2 off to simulate node downtime.

# systemctl poweroff 

Over time, Kubernetes migrated MySQL to k8s-node 1.

[root@k8s-master ~]# kubectl get pod
NAME                     READY   STATUS        RESTARTS   AGE
mysql-84bdf65dd5-bjz8b   1/1     Terminating   0          22m
mysql-84bdf65dd5-ddlhc   1/1     Running       0          34s

Verify data consistency:

[root@k8s-master ~]# kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword           If you don't see a command prompt, try pressing enter.

mysql> use mysql
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select * from my_id;
+------+
| id   |
+------+
|  111 |
+------+
1 row in set (0.01 sec)

mysql> quit

MySQL service is restored and the data is intact.

Volumes of emptyDir and hostPath types are convenient, but not durable. Kubernetes supports Volumes of various external storage systems.

PV and PVC separate the responsibilities of administrators and ordinary users, and are more suitable for production environment.