Environment Version Description:

  • Three VMware virtual machines, system version centos7.6.
  • Kubernetes 1.16.0, the latest version.
  • flannel v0.11
  • docker 18.09

Using kubeadm can simply build a k8s cluster environment without paying attention to the details of installation and deployment. Moreover, the k8s version update frequency is very fast, so this method is highly recommended.

Related preparation

Note: the related operations in this section should be performed on all nodes.

hardware environment

Three VMware virtual machines are used to configure the network and ensure the network can be connected.

  • K8s master 4G 4-core centos7
  • K8s-node1 2g 2 core centos7
  • K8s-node2 2g 2 core centos7

Host Division

  • K8s master as cluster management node: etcd kubeadm Kube apiserver Kube scheduler Kube Controller Manager kubelet flanneld docker
  • K8s-node1 as work node: kubeadm kubelet flanneld docker
  • K8s-node2 as work node: kubeadm kubelet flanneld docker


Install the necessary RPM software:

 yum install -y wget vim net-tools epel-release

Turn off the firewall

systemctl disable firewalld
systemctl stop firewalld

Shut down SELinux

#Temporarily disable SELinux
setenforce 0
#Permanently close to modify / etc / sysconfig / SELinux file settings
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

Disable swap partition

swapoff -a

#Disable it permanently. Open / etc / fstab and comment out the line swap.
sed -i 's/.*swap.*/#&/' /etc/fstab

Modify / etc / hosts

cat <<EOF >> /etc/host k8s-master k8s-node1 k8s-node2

Modifying kernel parameters

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
sysctl --system

Installing docker18.09

Configure Yum source

##Configure default source
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

##Download aliyuan
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

yum makecache fast

##Configure k8s source
cat <<EOF > /etc/yum.repos.d/kubernetes.repo

##Rebuild Yum cache
yum clean all
yum makecache fast
yum -y update

Install docker

Download docker’s Yum source file

yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

The docker version is specified here. You can view the supported versions first

[[email protected] ~]# yum list docker-ce --showduplicates |sort -r               
 * updates: mirrors.aliyun.com
Loading mirror speeds from cached hostfile
Loaded plugins: fastestmirror
 * extras: mirrors.aliyun.com
 * epel: hkg.mirror.rackspace.com
docker-ce.x86_64            3:19.03.2-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.1-3.el7                     docker-ce-stable
docker-ce.x86_64            3:19.03.0-3.el7                     docker-ce-stable
docker-ce.x86_64            3:18.09.9-3.el7                     docker-ce-stable
 * base: mirrors.aliyun.com

At present, the latest version is 19.03, and the specified download is 18.09

yum install -y docker-ce-18.09.9-3.el7
systemctl enable docker
systemctl start docker

Modify the starting parameters of docker

cat > /etc/docker/daemon.json <<EOF
  "registry-mirrors": ["https://xxxx.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  "storage-driver": "overlay2",
  "storage-opts": [

systemctl restart docker

Installation of k8s

Manage node configuration

First install the management node on the k8s master

Download kubeadm, kubelet

yum install -y kubeadm kubelet

Initialize kubeadm

There is no direct initialization, because domestic users can’t directly pull the relevant image, so here you want to view the required image version

[[email protected] ~]# kubeadm config images list

According to the required version, pull the domestic image directly and modify the tag

# vim kubeadm.sh


##Use the following script to download the domestic image and modify the tag to Google's tag
set -e




for imageName in ${images[@]} ; do
  docker pull $ALIYUN_URL/$imageName
  docker tag  $ALIYUN_URL/$imageName $GCR_URL/$imageName
  docker rmi $ALIYUN_URL/$imageName

Run the script and pull the image

sh ./kubeadm.sh

Master node execution

sudo kubeadm init \
 --apiserver-advertise-address \
 --kubernetes-version=v1.16.0 \

Note: the pod network CIDR here should not be changed, and it is related to the following steps.

Result return


mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

##To add a node, you need to execute the following command, which can be obtained by using the command kubeadm token create -- print join command 
kubeadm join --token lixsl8.v1auqmf91ty0xl0k \
    --discovery-token-ca-cert-hash sha256:c3f92a6ed9149ead327342f48a545e7e127a455d5b338129feac85893d918a55

If you want to install multiple master nodes, the initialization command uses the

kubeadm init  --apiserver-advertise-address --control-plane-endpoint  --kubernetes-version=v1.16.0  --pod-network-cidr=  --upload-certs

To add a master node, use the following command:

kubeadm join --token z34zii.ur84appk8h9r3yik --discovery-token-ca-cert-hash sha256:dae426820f2c6073763a3697abeb14d8418c9268288e37b8fc25674153702801     --control-plane --certificate-key 1b9b0f1fdc0959a9decef7d812a2f606faf69ca44ca24d2e557b3ea81f415afe

Note: do not copy the token directly. After kubeadm init succeeds, the command to add master node will be output

Work node configuration

Download kubedm kubelet

All three nodes need to run

yum install -y kubeadm kubelet

Add node

All three nodes need to run. The error is ignored here

kubeadm join --token lixsl8.v1auqmf91ty0xl0k \
    --discovery-token-ca-cert-hash sha256:c3f92a6ed9149ead327342f48a545e7e127a455d5b338129feac85893d918a55 \

If adding a node fails or you want to add it again, you can use the command

kubeadm reset

Note: do not use on easy master, it will delete all kubeadm configurations
At this time, you can use the command on the node to view the added node information

[[email protected] ~]# kubectl get nodes
k8s-master   NotReady   master   45m   v1.16.0
k8s-node1    NotReady   <none>   26s   v1.16.0
k8s-node2    NotReady   <none>   12s   v1.16.0

But the state of the node isNotReady, some more operations are required

Install flanneld

Operate on the master and copy the configuration to make kubectl available

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Download flannel configuration file

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Because of Kube- flannel.yml The image used in the file is quay.io It can’t be pulled in China, so download it from the domestic source first, and then modify the tag. The script is as follows

# vim flanneld.sh


set -e


#Modify the source here


for imageName in ${images[@]} ; do
  docker pull $QINIU_URL/$imageName
  docker tag  $QINIU_URL/$imageName $QUAY_URL/$imageName
  docker rmi $QINIU_URL/$imageName

Run the script, which needs to be executed on each node

sh flanneld.sh

Install flanneld

kubectl apply -f kube-flanneld.yaml

Flanneld is installed in the Kube system namespace by default. Use the following command to view it:

# kubectl -n kube-system get pods
NAME                                 READY   STATUS         RESTARTS   AGE
coredns-5644d7b6d9-h9bxt             0/1     Pending        0          57m
coredns-5644d7b6d9-pkhls             0/1     Pending        0          57m
etcd-k8s-master                      1/1     Running        0          57m
kube-apiserver-k8s-master            1/1     Running        0          57m
kube-controller-manager-k8s-master   1/1     Running        0          57m
kube-flannel-ds-amd64-c4hnf          1/1     Running        1          38s
kube-flannel-ds-amd64-djzmx          1/1     Running        0          38s
kube-flannel-ds-amd64-mdg8b          1/1     Running        1          38s
kube-flannel-ds-amd64-tjxql          0/1     Terminating    0          5m34s
kube-proxy-4n5dr                     0/1     ErrImagePull   0          13m
kube-proxy-dc68d                     1/1     Running        0          57m
kube-proxy-zplgt                     0/1     ErrImagePull   0          13m
kube-scheduler-k8s-master            1/1     Running        0          57m

An error occurred because the two work nodes cannot pull the pause and cube proxy images. They can be packaged directly from the master and used on the node

##Execute on master
docker save -o pause.tar k8s.gcr.io/pause:3.1
docker save -o kube-proxy.tar k8s.gcr.io/kube-proxy

##Execute on node
docker load -i pause.tar 
docker load -i kube-proxy.tar

Re install flanneld

 kubectl delete -f kube-flannel.yml 
 kubectl create -f kube-flannel.yml 

Modify kubelet

After adding node with kubeadm, the node is always in theNotReadyStatus, error message is:

runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Addresses:

The solution is to modify / var / lib / kubelet / kubeadm- flags.env Files, deleting parameters--network-plugin=cni

cat << EOF > /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --pod-infra-container-image=k8s.gcr.io/pause:3.1"

systemctl restart kubelet
[[email protected] ~]# kubectl get nodes
k8s-master   Ready    master   79m   v1.16.0
k8s-node1    Ready    <none>   34m   v1.16.0
k8s-node2    Ready    <none>   34m   v1.16.0

Error resolution

About mistakescni config uninitialized AddressesPreviously, the parameter was deleted directly--network-plugin=cniHowever, this can only change the node status to ready, but the network is still unavailable between pods.

The right solution: modify Kube- flannel.yaml , add the parameter on line 111cniVersion

vim kube-flanneld.yaml

      "name": "cbr0",
      "cniVersion": "0.3.1",

Install flannel

##If it was installed before, remove it first
## kubectl delete -f kube-flannel.yaml

kubectl apply -f kube-flannel.yaml