kubevirt入门

1. 前置条件

1.1 配置本地存储

# local path-provisioner # https://github.com/rancher/local-path-provisioner
wget https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.29/deploy/local-path-storage.yaml
sed -i 's#/opt/local-path-provisioner#/opt/k8s-local-path-storage#g'
crictl pull dhub.kubesre.xyz/rancher/local-path-provisioner:v0.0.29
ctr -n k8s.io i tag dhub.kubesre.xyz/rancher/local-path-provisioner:v0.0.29 docker.io/rancher/local-path-provisioner:v0.0.29
k apply -f local-path-storage.yaml

1.2 开启特性门控

# 开启StatefulSetAutoDeletePVC特性门控
# 此门控在1.27已经默认启用
sudo vim /etc/kubernetes/manifests/kube-controller-manager.yaml
# 添加参数
- --feature-gates=StatefulSetAutoDeletePVC=true
# 关闭保存后kube-controller-manager会自动重新加载

2. 安装依赖

sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager

3. 检查是否支持虚拟化

virt-host-validate qemu

4. 安装kubevirt

# 安装kubevirt
export RELEASE=v1.3.1
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml
# 等待所有 KubeVirt 组件都启动
kubectl -n kubevirt wait kv kubevirt --for condition=Available

# 下载 virtctl client
wget https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/virtctl-${RELEASE}-linux-amd64
mv virtctl-${RELEASE}-linux-amd64 /usr/local/bin/virtctl
chmod +x /usr/local/bin/virtctl

# 安装CDI：用于将pvc转换为虚拟机的磁盘
export RELEASE=v1.60.3
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$RELEASE/cdi-operator.yaml
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$RELEASE/cdi-cr.yaml

# 启用featureGates
vim enable-feature-gate.yaml
# 写入如下内容
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  configuration:
    vmRolloutStrategy: "LiveUpdate"
    developerConfiguration:
      featureGates:
        - ExpandDisks
        - HotplugVolumes
        - VMLiveUpdateFeatures
  workloadUpdateStrategy:
    workloadUpdateMethods:
    - LiveMigrate

# 应用更改
kubectl apply -f enable-feature-gate.yaml

5. 编写部署helm文件

# templates/NOTES.txt
{{ include "ubuntu.fullname" . }} is deployed!
Before start, you must wait it start successfully.
You can run `kubectl get dv | grep {{ include "ubuntu.fullname" . }}` to check if the image has been pulled.
You can run `virtctl console {{ include "ubuntu.fullname" . }}` to connect to it.
You can type ctrl+] to exit console.
The username is {{ .Values.software.user }} and password is {{ .Values.software.password }}.
{{- if .Values.software.desktop.enabled }}
Please wait the desktop environment installed, then you can run `virtctl vnc {{ include "ubuntu.fullname" . }}` to connect to it.
You can see `/var/log/cloud-init-output.log` to see the progress of installing.
{{- end }}
Have a happy coding in {{ include "ubuntu.fullname" . }}!


# templates/datavolume.yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: {{ include "ubuntu.fullname" . }}
spec:
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: {{ .Values.hardware.storage }}
    storageClassName: local-path
  source:
    http:
      {{- if eq .Values.software.customImageUrl ""}}
      url: "https://cloud-images.ubuntu.com/{{ .Values.software.ubuntuVersion }}/current/{{ .Values.software.ubuntuVersion }}-server-cloudimg-amd64.img"
      {{- else }}
      url: {{ .Values.software.customImageUrl }}
      {{- end }}


# templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: {{ include "ubuntu.fullname" . }}-cloudinit-nocloud
type: Opaque
stringData:
  userdata: |-
    #cloud-config
    user: "{{ .Values.software.user }}"
    password: "{{ .Values.software.password }}"
    chpasswd: { expire: False }
    ssh_pwauth: True
    {{- if .Values.software.sshConfig.enabled }}
    write_files:
      - path: /home/root/ssh/{{ .Values.software.sshConfig.idFile }}
        content: |
        {{- .Values.software.sshConfig.idFileContent | nindent 18 }}
      - path: /home/root/ssh/{{ .Values.software.sshConfig.idFile }}.pub
        content: |
        {{- .Values.software.sshConfig.idFilePubContent | nindent 18 }}
    {{- end }}
    runcmd:
      - sed -i s@/archive.ubuntu.com/@/mirrors.tuna.tsinghua.edu.cn/@g /etc/apt/sources.list
      - sh -c '[ -f /etc/apt/sources.list.d/ubuntu.sources ] && sed -i -E "s|http://archive\.ubuntu\.com/ubuntu/?|http://cn.archive.ubuntu.com/ubuntu/|g" /etc/apt/sources.list.d/ubuntu.sources'
      - apt-get update
      - apt-get install zsh -y
      - wget https://gitee.com/mirrors/oh-my-zsh/raw/master/tools/install.sh
      - sed -i 's/REPO=\${REPO:-ohmyzsh\/ohmyzsh}/REPO=\${REPO:-mirrors\/oh-my-zsh}/g' install.sh
      - sed -i 's/REMOTE=\${REMOTE:-https:\/\/github.com\/\${REPO}.git}/REMOTE=\${REMOTE:-https:\/\/gitee.com\/\${REPO}.git}/g' install.sh
      - sed -i 's/USER=\${USER:-\$(id -u -n)}/USER={{ .Values.software.user }}/' install.sh
      - bash install.sh
      - rm install.sh
      - sed -i 's/ZSH_THEME="robbyrussell"/ZSH_THEME="ys"/g' /home/{{ .Values.software.user }}/.zshrc
      - sed -i 's/plugins=(git)/plugins=(git zsh-syntax-highlighting zsh-autosuggestions)/g' /home/{{ .Values.software.user }}/.zshrc
      - git clone https://gitee.com/mirrors/zsh-syntax-highlighting.git /home/{{ .Values.software.user }}/.oh-my-zsh/custom/plugins/zsh-syntax-highlighting
      - git clone https://gitee.com/mirrors/zsh-autosuggestions.git /home/{{ .Values.software.user }}/.oh-my-zsh/custom/plugins/zsh-autosuggestions
      - echo "exec -l zsh" >> /home/{{ .Values.software.user }}/.bashrc
    {{- if .Values.software.sshConfig.enabled }}
      - cp /home/root/ssh/{{ .Values.software.sshConfig.idFile }} /home/{{ .Values.software.user }}/.ssh/{{ .Values.software.sshConfig.idFile }}
      - cp /home/root/ssh/{{ .Values.software.sshConfig.idFile }}.pub /home/{{ .Values.software.user }}/.ssh/{{ .Values.software.sshConfig.idFile }}.pub
      - rm -rf /home/root/ssh
      - chown {{ .Values.software.user }}:{{ .Values.software.user }} /home/{{ .Values.software.user }}/.ssh/*
      - chmod 0600 /home/{{ .Values.software.user }}/.ssh/*
    {{- end }}
    {{- if .Values.software.desktop.enabled }}
      {{- if .Values.software.desktop.fullInstalled }}
      - DEBIAN_FRONTEND=noninteractive apt-get install ubuntu-desktop -y
      {{- else }}
      - DEBIAN_FRONTEND=noninteractive apt-get install ubuntu-desktop-minimal -y
      {{- end }}
      - DEBIAN_FRONTEND=noninteractive apt-get install lightdm debconf-utils -y
      - DEBIAN_FRONTEND=noninteractive apt-get remove gdm3 -y
      - DEBIAN_FRONTEND=noninteractive dpkg-reconfigure lightdm
      - service lightdm start
      - cat /etc/X11/default-display-manager
    {{- end }}
      - echo "Cloud init done."


# templates/vm.yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/os: linux
  name: {{ include "ubuntu.fullname" . }}
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/domain: {{ include "ubuntu.fullname" . }}
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: disk0
          - disk:
              bus: virtio
              readonly: true
            name: cloudinitdisk
        machine:
          type: ""
        resources:
          requests:
            cpu: {{ .Values.hardware.cpu }}
            memory: {{ .Values.hardware.memory }}
      terminationGracePeriodSeconds: 0
      volumes:
      - name: disk0
        persistentVolumeClaim:
          claimName: {{ include "ubuntu.fullname" . }}
      - cloudInitNoCloud:
          secretRef:
            name: {{ include "ubuntu.fullname" . }}-cloudinit-nocloud
        name: cloudinitdisk


# values.yaml
# Default values for ubuntu.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# This is configs for hardware

hardware:
  storage: 40Gi
  cpu: 4
  memory: 8096M

# This is configs for software
software:
  user: ethereal
  password: "020402"
  ubuntuVersion: jammy
  # ubuntuVersion: focal
  customImageUrl: ""
  desktop:
    enabled: false
    fullInstalled: false
  sshConfig:
    enabled: false
    idFile: id_ed25519
    idFileContent: |
      <私钥>
    idFilePubContent: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID7HfYBgf1NssgJHQzmdsMaABb+qAtHRULU3p/91sQWd ethereal@Ethereal-Desktop

6. 附加额外内容

可以附加硬盘
添加如下helm文件

# values.yaml
# Default values for addvolume.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# This is configs for hardware

hardware:
  storage: 20Gi


# templates/datavolume.yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: {{ include "addvolume.fullname" . }}
spec:
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: {{ .Values.hardware.storage }}
    storageClassName: local-path
  source:
    blank: {}

# templates/NOTES.txt
{{ include "addvolume.fullname" . }} is deployed!
You can run `kubectl get dv | grep {{ include "addvolume.fullname" . }}` to check the dv status.
You can attach it to a vm by `virtctl addvolume vm_name --volume-name={{ include "addvolume.fullname" . }} --persist`

可以使用如下方法附加

helm install ubuntu-vm-2204-vm2 ~/Softwares/k8s-help-tools/vm/addvolume
virtctl addvolume ubuntu-vm-2204 --volume-name=ubuntu-vm-2204-vm2 --persist

可以使用如下方法修改cpu和内存

# 停止虚拟机
virtctl stop ubuntu-vm-2204
# 修改cpu或内存配置
vim values.yaml
# 更新
helm upgrade ubuntu-vm-2204 ~/Softwares/k8s-help-tools/vm/myubuntu --set software.ubuntuVersion=jammy --set software.desktop.enabled=true --set software.sshConfig.enabled=true

7. 特殊情况

当vm关机后开启时，网卡无法启动，原因是netplan中指定了网卡的mac地址，而mac地址在关机开机后发生改变

首先进入文件夹/etc/netplan/，发现其中文件（50-cloud-init.yaml）内容为：

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        enp1s0:
            dhcp4: true
            dhcp6: true
            match:
              macaddress: b2:6a:f3:79:c0:47
            set-name: enp1s0
    version: 2

其中指明了mac地址。

因此只需要将mac地址修改为ip a所显示的mac地址，保存后使用命令sudo netplan apply部署即可。

8. 救砖

通过管理文件即可救砖。

## 通过外带的机器挂载镜像
virt-rescue -a CentOS_8_Server.img -i
## 修改密码
virt-sysprep -d 21admin --firstboot-command ‘dpkg-reconfigure openssh-server’ --password ubuntu:password:123456 --root-password password:123456
## 工具安装
apt/yum install libguestfs-tools 
systemctl start libvirtd
## 详细信息
virt-sysprep -v -x ...

9. kube-ovn

kube-ovn是kubevirt的最佳组合，同时，kube-ovn支持通过chaining模式与cilium结合，使得同时具备高性能和易用性。

9.0 可选：删除之前的cni

# master
cilium uninstall
​
# all nodes
ip a | grep cili | awk -F '{print $2}' | awk -F '@' '{print $1}' | xargs -i ip link del {}
ip link del kube-ipvs0
rm -f /etc/cni/net.d/*
​
systemctl restart kubelet
systemctl restart network

sudo reboot # 重启机器可以删干净规则

9.1 搭建kube-ovn

# kube-ovn
wget https://github.com/kubeovn/kube-ovn/releases/download/v1.13.5/kube-ovn-v1.13.5.tgz
tar xvf kube-ovn-v1.13.5.tgz

# 给节点添加标签
kubectl label node -lbeta.kubernetes.io/os=linux kubernetes.io/os=linux --overwrite
kubectl label node -lnode-role.kubernetes.io/control-plane kube-ovn/role=master --overwrite

# 更改helm文件
MASTER_NODES: "192.168.29.100"
ipv4:
  POD_CIDR: "10.244.0.0/16"
  POD_GATEWAY: "10.244.0.1"
  SVC_CIDR: "10.96.0.0/12"
  JOIN_CIDR: "100.64.0.0/16"
  PINGER_EXTERNAL_ADDRESS: "1.1.1.1"
  PINGER_EXTERNAL_DOMAIN: "kube-ovn.io."

ipv6:
  POD_CIDR: "fd00:10:244::/112"
  POD_GATEWAY: "fd00:10:244::1"
  SVC_CIDR: "fd00:10:96::/112"
  JOIN_CIDR: "fd00:100:64::/112"
  PINGER_EXTERNAL_ADDRESS: "2606:4700:4700::1111"
  PINGER_EXTERNAL_DOMAIN: "google.com."

dual_stack:
  POD_CIDR: "10.244.0.0/16,fd00:10:244::/112"
  POD_GATEWAY: "10.244.0.1,fd00:10:244::1"
  SVC_CIDR: "10.96.0.0/12,fd00:10:96::/112"
  JOIN_CIDR: "100.64.0.0/16,fd00:100:64::/112"
  PINGER_EXTERNAL_ADDRESS: "1.1.1.1,2606:4700:4700::1111"
  PINGER_EXTERNAL_DOMAIN: "google.com."


# 如果需要结合cilium，还需要修改
func:
  ENABLE_NP: false
cni_conf:
  CNI_CONFIG_PRIORITY: "10"


# 部署
helm install kube-ovn ./

9.2 结合cilium

删除之前的安装

cilium uninstall

调整kube-ovn的配置

func:
  ENABLE_NP: false
cni_conf:
  CNI_CONFIG_PRIORITY: "10"

每个节点调整cni优先级

mv /etc/cni/net.d/01-kube-ovn.conflist /etc/cni/net.d/10-kube-ovn.conflist

创建chaining.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cni-configuration
  namespace: kube-system
data:
  cni-config: |-
    {
      "name": "generic-veth",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "kube-ovn",
          "server_socket": "/run/openvswitch/kube-ovn-daemon.sock",
          "ipam": {
              "type": "kube-ovn",
              "server_socket": "/run/openvswitch/kube-ovn-daemon.sock"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "cilium-cni"
        }
      ]
    }

安装chaining文件

kubectl apply -f chaining.yaml

重新部署cilium

# 注意下列CIDR要和上面kube-ovn安装时候的保持一致
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --wait  --version 1.16.1        --namespace kube-system         --set operator.replicas=1       --set socketLB.enabled=true     --set nodePort.enabled=true     --set externalIPs.enabled=true  --set hostPort.enabled=false --set routingMode=native        --set sessionAffinity=true      --set enableIPv4Masquerade=false        --set enableIPv6Masquerade=false        --set hubble.enabled=true       --set sctp.enabled=true         --set ipv4.enabled=true         --set ipv6.enabled=false        --set ipam.mode=cluster-pool    --set-json ipam.operator.clusterPoolIPv4PodCIDRList='["100.65.0.0/16"]'         --set-json ipam.operator.clusterPoolIPv6PodCIDRList='["fd00:100:65::/112"]'     --set cni.chainingMode=generic-veth     --set cni.chainingTarget=kube-ovn       --set cni.customConf=true       --set cni.configMap=cni-configuration

查看部署状态，可以看到有多少pod正在被管理

cilium  status

重启机器，更新ip link

sudo reboot

重启所有pod

kubectl delete pods --all -n <namespace>

如果前后服务的cidr一致，那么则不需要重新安装服务。

9.3 安装插件

# 下载插件
wget https://raw.githubusercontent.com/kubeovn/kube-ovn/release-1.10/dist/images/kubectl-ko
# 复制到运行目录
cp kubectl-ko /usr/local/bin/kubectl-ko
# 添加执行权限
chmod +x /usr/local/bin/kubectl-ko
# 检查是否可以正常使用
kubectl plugin list
# 示例命令
kubectl ko nb status

命令参考kubectl 插件使用 - Kube-OVN 文档

10. 架构

10.1 kube-ovn架构

10.1.1 来自上游：ovn-central

运行 OVN 的管理平面组件，包括 ovn-nb, ovn-sb, 和 ovn-northd。北向数据库持有的是物理抽象，例如路由器、网桥等；南向数据库持有的是流表，可以通过以下命令查看：

kubectl ko nbctl list Logical_Switch
kubectl ko sbctl list Logical_Flow

多个 ovn-central 实例会通过 Raft 协议同步数据保证高可用。

10.1.2 来自上游：ovs-ovn

以 DaemonSet 形式运行在每个节点，在 Pod 内运行了 openvswitch, ovsdb, 和 ovn-controller。这些组件作为 ovn-central 的 Agent 将逻辑流表翻译成真实的网络配置。类似于cilium中的cilium组件。当与cilium链式配合时，改用cilium创建与控制将逻辑流表翻译成真实的网络配置。

主要是用于service层，也就是说从pod网络到其他pod网络的连通性。

10.1.3 核心控制器：kube-ovn-controller

监听k8s组件所有和网络功能相关资源的事件，并根据资源变化情况更新 OVN 内的逻辑网络。也就是说，将k8s的网络逻辑翻译成ovn的网络逻辑，写入北向数据库

10.1.4 核心控制器：kube-ovn-cni

运行在每个节点上，实现 CNI 接口，并操作本地的 OVS 配置单机网络。

主要是负责部署从pod的clusterIP连接到pod的网络

10.1.5 扩展组件：kube-ovn-speaker

对外发布pod的clusterIP，使得能够在本地访问

10.1.6 扩展组件：kube-ovn-pinger

运行在每个节点上收集 OVS 运行信息，节点网络质量，网络延迟等信息。

10.1.7 扩展组件：kube-ovn-monitor

收集 OVN 的运行信息。

10.1.8 例子

举一个创建 Pod 的例子，Pod 下发到 apiserver 后 kube-ovn-controller 会 watch 的新生成了一个 Pod，然后调用 ovn-nb 的接口去创建一个虚拟交换机接口，成功后将 OVN 分配的 IP、Mac、GW 等信息反写到这个 Pod 的 Annotation 中。接下来 kubelet 创建 Pod 时会调用 kube-ovn-cni，kube-ovn-cni 将信息传递给 kube-ovn-cniserver。CNIServer 会反查 Pod 的 Annotation 获得具体的 IP 和 Mac 信息来配置本地的 OVS 和容器网卡，完成整个工作流。其他的 Service、NetworkPolicy 的流程也是和这个类似的。

10.2 kubevirt架构

10.2.1 virt-operator

通过kubevirt operator安装kubevirt相关组件，选择指定版本。

10.2.2 virt-api

提供 HTTP RESTful 接口来管理集群内的虚拟机和虚拟机相关的工作流，负责更新虚拟化 CRD，也负责 VMI CRD 的默认设置和验证。它是一个集群级别的组件。

10.2.3 virt-controller

负责集群范围的虚拟化功能，管理与 VMI 关联的 Pod 的生命周期。它是一个集群级别的组件。

10.2.4 virt-handler

在各个节点上运行的 DaemonSet 资源，用于监控 VM 对象的更改，以及执行必要的操作来达到所需状态。

10.2.5 virt-launcher

VMI 相关的 Pod 中包含了一个 virt-launcher 组件，用于提供 CGroup 和命名空间来托管 VMI 进程，当 VM CRD 对象通过 virt-handler 组件传递给 virt-launcher 时，将使用 Pod 中的 libvirtd 容器来启动 VMI，virt-launcher 将监控 VMI 进程，直至 VMI 终止时退出。

需要注意的是，后续附加的磁盘也是运行的virt-launcher。

10.3 cdi

10.3.1 cdi-operator

负责创建cdi。

10.3.2 cdi-uploadproxy

负责创建从本地导入的dataVolume。

10.3.3 cdi-deployment

包含CDI Controller，负责创建dataVolume，当导入时会创建一个Data Import Pod。

10.3.4 cdi-apiserver

提供外部访问的接口。

11. 其他

11.1 cloud-init日志

# log文件
## 普通log
/var/log/cloud-init-output.log: Captures the output from each stage of cloud-init when it runs.
## 详细log
/var/log/cloud-init.log: Very detailed log with debugging output, describing each action taken.

# Early boot log
## 检查没有成功运行原因
/run/cloud-init/cloud-init-generator.log: On systemd systems, this log file describes early boot enablement of cloud-init via the systemd generator. These logs are most useful if trying to figure out why cloud-init did not run.
## 检查cloud init没有识别
/run/cloud-init/ds-identify.log: Contains logs about platform / datasource detection. These logs are most useful if cloud-init did not identify the correct datasource (cloud) to run on.

# 配置文件
/etc/cloud/cloud.cfg
/etc/cloud/cloud.cfg.d/*.cfg