离线Ingress网关离线部署、内网反向代理闭环搭建技术栈Kubernetes v1.32.13 Rocky Linux 8.6 Containerd 1.7.x操作环境 / 对接原理 / 详细步骤 / 完整命令 / 配置文件 / 验证流程 / 排错方案离线Ingress网关离线部署、内网反向代理闭环搭建操作环境K8s集群3节点k8s-master(192.168.1.10), k8s-node1(192.168.1.11), k8s-node2(192.168.1.12)K8s版本 v1.32.13容器运行时 Containerd 1.7.x操作系统 Rocky Linux 8.6离线环境无外网物理隔离机房内网网段 192.168.1.0/24已搭建离线YUM源、离线镜像仓库(Harbor)、离线NTP服务离线场景K8s离线集群从零部署搭建已完成离线环境基础搭建K8s集群已部署离线镜像仓库已对接离线监控日志系统已部署存储配置本地存储/NFS/Ceph数据目录 /data已配置StorageClass和PVC动态供给已配置离线监控系统PrometheusGrafana可查看集群指标、节点指标、应用指标、日志、事件便于监控和排障对接原理离线Ingress网关离线部署、内网反向代理闭环搭建是 K8s 离线环境专属对接场景中的核心操作场景。离线环境是政企内网、涉密机房、私有化部署的典型形态无外网物理隔离带来了软件包获取、镜像拉取、漏洞库更新、时间同步、日志归集等一系列特殊挑战。K8s离线集群从零部署搭建作为离线环境运维的具体领域负责离线环境规划预检、K8s集群离线部署、离线镜像仓库运维、离线网络适配、离线存储对接、离线监控日志告警、离线CI/CD持续交付等核心能力。离线环境运维的核心目标是在无外网条件下实现K8s集群的稳定运行、安全可控、可追溯、可演进通过前置规划和环境预检确保离线环境基础达标通过离线安装包和依赖归档实现K8s集群从零部署通过离线镜像仓库和镜像全生命周期管理实现容器镜像的可控分发通过离线网络深度适配实现集群内外网络连通与隔离通过离线存储全场景适配实现数据持久化与安全通过离线监控日志告警体系实现无外网可观测通过离线CI/CD实现业务持续交付。所有操作需遵循先规划后实施、先备份后操作、先验证后上线的安全原则避免离线环境操作引发集群中断。详细步骤1. 离线环境现状盘点与基础状态检查# 1. 检查离线环境基础状态 ssh root192.168.1.10 # 检查系统信息 cat /etc/os-release uname -r # 检查网络 ip addr ip route ping -c 3 192.168.1.11 # 检查时间 timedatectl chronyc sources 2/dev/null || ntpq -p 2/dev/null # 检查主机名和hosts hostname cat /etc/hosts # 检查防火墙 firewall-cmd --state 2/dev/null firewall-cmd --list-ports 2/dev/null iptables -L -n | head -20 # 检查SELinux getenforce sestatus # 检查swap free -h swapon --show # 检查磁盘 df -h lsblk # 检查内核模块 lsmod | grep -E overlay|br_netfilter|ip_vs # 检查系统参数 sysctl net.ipv4.ip_forward sysctl net.bridge.bridge-nf-call-iptables sysctl net.bridge.bridge-nf-call-ip6tables # 2. 检查K8s集群状态 kubectl get nodes -o wide kubectl get pods -A -o wide kubectl get cs # 检查组件状态 kubectl get pods -n kube-system # 检查集群信息 kubectl cluster-info kubectl version # 3. 检查容器运行时 systemctl status containerd containerd --version ctr version # 检查containerd配置 cat /etc/containerd/config.toml | head -50 # 检查镜像 ctr images ls | head -20 # 4. 检查离线YUM源 cat /etc/yum.repos.d/*.repo yum repolist # 测试安装 yum list available | head -20 # 5. 检查离线镜像仓库 curl -k -I https://harbor.example.com 2/dev/null curl -k -u admin:${HARBOR_PASSWORD} https://harbor.example.com/api/v2.0/projects 2/dev/null | jq .[].name # 检查containerd仓库配置 cat /etc/containerd/config.toml | grep -A10 registry.mirrors # 6. 检查离线NTP chronyc tracking 2/dev/null chronyc sources -v 2/dev/null # 检查时间同步 timedatectl status # 7. 检查离线监控 curl -s http://prometheus:9090/api/v1/status/config 2/dev/null | head -20 curl -s http://grafana:3000/api/health 2/dev/null # 检查节点指标 kubectl top nodes kubectl top pods -A # 8. 导出当前配置备份 mkdir -p /tmp/offline_backup_$(date %Y%m%d) cp -r /etc/containerd /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null cp -r /etc/kubernetes /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null cp /etc/hosts /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null cp /etc/yum.repos.d/*.repo /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null echo 配置已备份到 /tmp/offline_backup_$(date %Y%m%d)/ 2. 制定操作方案与备份防护# 1. 制定离线环境操作方案 cat /tmp/offline_operation_plan.md EOF # 离线环境操作方案 ## 一、操作目标 ## 二、影响范围评估 ## 三、操作步骤与时间窗口 ## 四、回滚方案 ## 五、验证标准 ## 六、风险点与应对措施 ## 七、离线环境特殊注意事项 EOF echo 操作方案模板已创建 # 2. 确认离线资源可用性 # 检查离线安装包 ls -lh /opt/offline/packages/ 2/dev/null # 检查离线镜像 ls -lh /opt/offline/images/ 2/dev/null # 检查离线YUM源 df -h /var/www/html/yum/ 2/dev/null # 检查离线镜像仓库容量 df -h /data/harbor/ 2/dev/null # 3. 备份关键数据 # 备份K8s配置 kubectl get all -A -o yaml /tmp/k8s_all_backup_$(date %Y%m%d).yaml # 备份etcd kubectl -n kube-system exec etcd-k8s-master -- etcdctl snapshot save /tmp/etcd_snapshot_$(date %Y%m%d).db --endpointshttps://127.0.0.1:2379 --cacert/etc/kubernetes/pki/etcd/ca.crt --cert/etc/kubernetes/pki/etcd/server.crt --key/etc/kubernetes/pki/etcd/server.key 2/dev/null # 备份Harbor数据 cd /opt/harbor docker exec harbor-db pg_dump -U postgres registry /tmp/harbor_db_backup_$(date %Y%m%d).sql 2/dev/null # 备份配置文件 tar czf /tmp/offline_config_backup_$(date %Y%m%d).tar.gz /etc/kubernetes /etc/containerd /etc/hosts /etc/yum.repos.d 2/dev/null echo 关键数据已备份 # 4. 通知相关业务方 # echo 离线环境运维操作通知 | mail -s 离线环境通知 adminexample.com echo 建议在业务低峰期执行离线环境操作 # 5. 确认业务窗口 echo 当前时间: $(date) echo 离线环境操作需谨慎避免影响业务 3. 执行离线环境配置操作# 1. 检查离线环境配置文件 # 检查K8s配置 ls -la /etc/kubernetes/ cat /etc/kubernetes/admin.conf | head -20 # 检查kubelet配置 cat /var/lib/kubelet/config.yaml | head -30 # 检查kube-proxy配置 cat /var/lib/kube-proxy/config.conf 2/dev/null | head -30 # 2. 检查容器运行时配置 cat /etc/containerd/config.toml # 检查cgroup驱动 grep SystemdCgroup /etc/containerd/config.toml # 检查镜像仓库配置 grep -A20 registry /etc/containerd/config.toml # 3. 检查CNI配置 ls -la /etc/cni/net.d/ cat /etc/cni/net.d/10-calico.conflist 2/dev/null | head -30 cat /etc/cni/net.d/10-flannel.conflist 2/dev/null | head -30 # 检查Calico kubectl get pods -n kube-system | grep calico kubectl get ippools 2/dev/null # 4. 检查CoreDNS kubectl get pods -n kube-system | grep coredns kubectl get configmap coredns -n kube-system -o yaml | head -40 # 测试DNS kubectl run dns-test --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default 2/dev/null # 5. 检查存储配置 kubectl get storageclass kubectl get pv kubectl get pvc -A # 检查本地存储 ls -la /data/ # 检查NFS showmount -e localhost 2/dev/null mount | grep nfs # 6. 检查监控配置 kubectl get pods -n monitoring 2/dev/null kubectl get svc -n monitoring 2/dev/null # 检查Prometheus配置 kubectl get configmap prometheus-server -n monitoring -o yaml 2/dev/null | head -40 # 7. 执行具体离线环境配置操作根据标题调整 echo 执行离线环境具体配置操作... echo 请根据操作方案执行具体步骤 # 8. 应用配置变更 # 修改配置后重启相关服务 # systemctl restart kubelet # systemctl restart containerd # kubectl rollout restart deployment coredns -n kube-system # 9. 等待服务重启完成 sleep 30 kubectl get nodes kubectl get pods -A | grep -v Running | grep -v Completed echo 配置变更已应用 4. 验证集群网络存储镜像对接# 1. 验证K8s集群状态 kubectl get nodes -o wide # 预期所有节点 Ready kubectl get pods -A -o wide # 预期所有Pod Running或Completed # 2. 验证节点资源 kubectl top nodes # 预期CPU和内存使用率正常 kubectl top pods -A # 预期Pod资源使用正常 # 3. 验证网络连通性 # 测试Pod间通信 kubectl run net-test1 --imagebusybox --restartNever -n default -- sleep 3600 kubectl run net-test2 --imagebusybox --restartNever -n default -- sleep 3600 sleep 10 POD1_IP$(kubectl get pod net-test1 -o jsonpath{.status.podIP}) kubectl exec net-test2 -- ping -c 3 $POD1_IP # 预期ping通 kubectl delete pod net-test1 net-test2 --ignore-not-found # 4. 验证Service和DNS kubectl run dns-test --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default # 预期DNS解析成功 kubectl run dns-test2 --imagebusybox --restartNever --rm -it -- wget -q -O- https://kubernetes.default.svc.cluster.local 2/dev/null # 预期Service访问成功 # 5. 验证容器运行时 systemctl status containerd # 预期active (running) ctr version # 预期版本正确 ctr images ls | head -10 # 预期镜像列表正常 # 6. 验证离线镜像仓库 curl -k -I https://harbor.example.com # 预期返回200或302 # 测试镜像推拉 docker login harbor.example.com -u admin -p ${HARBOR_PASSWORD} docker pull nginx:latest docker tag nginx:latest harbor.example.com/library/nginx:test docker push harbor.example.com/library/nginx:test docker rmi harbor.example.com/library/nginx:test docker pull harbor.example.com/library/nginx:test # 预期推拉成功 # 7. 验证K8s从离线仓库拉取镜像 kubectl run test-harbor --imageharbor.example.com/library/nginx:test --restartNever sleep 10 kubectl get pods test-harbor # 预期Pod Running kubectl delete pod test-harbor --ignore-not-found # 8. 验证存储 # 创建测试PVC cat EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi EOF sleep 10 kubectl get pvc test-pvc # 预期Bound kubectl delete pvc test-pvc --ignore-not-found # 9. 验证监控 kubectl get pods -n monitoring 2/dev/null # 预期监控Pod Running # 检查指标 curl -s http://prometheus:9090/api/v1/query?queryup 2/dev/null | jq .data.result | length # 预期返回指标数量 # 10. 清理测试资源 docker rmi harbor.example.com/library/nginx:test 2/dev/null echo 验证完成 5. 离线YUM/NTP/镜像仓库/监控配置# 1. 配置离线YUM源 # 创建本地YUM源目录 mkdir -p /var/www/html/yum/rocky8 # 同步软件包在有网机器执行 # reposync -g -l -d -m --repoidbaseos --newest-only --download-metadata -p /var/www/html/yum/rocky8/ # 离线环境直接使用已同步的源 cat /etc/yum.repos.d/offline.repo EOF [offline-baseos] nameOffline BaseOS baseurlfile:///var/www/html/yum/rocky8/baseos enabled1 gpgcheck0 EOF yum clean all yum repolist # 2. 配置离线NTP # 安装chrony yum install -y chrony # 配置内网NTP服务器 cat /etc/chrony.conf EOF server 192.168.1.10 iburst allow 192.168.1.0/24 local stratum 10 EOF systemctl enable --now chronyd chronyc sources -v # 3. 配置离线镜像仓库 # Harbor已部署配置containerd对接 cat /etc/containerd/config.toml EOF version 2 [plugins.io.containerd.grpc.v1.cri.registry.mirrors.harbor.example.com] endpoint [https://harbor.example.com] [plugins.io.containerd.grpc.v1.cri.registry.configs.harbor.example.com.tls] ca_file /etc/containerd/certs.d/harbor.example.com/ca.crt EOF # 导入证书 mkdir -p /etc/containerd/certs.d/harbor.example.com cp /opt/harbor/ssl/ca.crt /etc/containerd/certs.d/harbor.example.com/ systemctl restart containerd # 4. 配置离线监控 # Prometheus配置 cat /etc/prometheus/prometheus.yml EOF global: scrape_interval: 15s scrape_configs: - job_name: kubernetes-nodes kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token EOF # 5. 配置离线日志 # 配置日志轮转 cat /etc/logrotate.d/k8s EOF /var/log/k8s/*.log { daily rotate 30 compress delaycompress missingok notifempty create 0644 root root } EOF # 6. 配置离线安全基线 # 关闭swap swapoff -a sed -i /swap/d /etc/fstab # 关闭SELinux setenforce 0 sed -i s/^SELINUX.*/SELINUXdisabled/ /etc/selinux/config # 配置内核参数 cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl --system # 7. 配置离线CI/CD # Jenkins配置 # 管理Jenkins插件离线安装 # 配置Jenkins任务对接离线镜像仓库 echo 离线CI/CD配置完成 6. 最终验证与操作报告# 1. 最终验证K8s集群状态 kubectl get nodes -o wide # 预期所有节点 Ready kubectl get pods -A | grep -v Running | grep -v Completed # 预期无异常Pod # 2. 最终验证离线YUM源 yum repolist # 预期离线源可用 yum list available | wc -l # 预期返回软件包数量 # 3. 最终验证离线NTP chronyc tracking # 预期时间同步正常 timedatectl status # 预期时间正确 # 4. 最终验证离线镜像仓库 curl -k -I https://harbor.example.com # 预期返回200或302 docker login harbor.example.com -u admin -p ${HARBOR_PASSWORD} docker pull nginx:latest docker tag nginx:latest harbor.example.com/library/nginx:verify docker push harbor.example.com/library/nginx:verify docker rmi harbor.example.com/library/nginx:verify docker pull harbor.example.com/library/nginx:verify # 预期推拉成功 # 5. 最终验证K8s对接离线仓库 kubectl run verify-offline --imageharbor.example.com/library/nginx:verify --restartNever sleep 10 kubectl get pods verify-offline # 预期Pod Running kubectl delete pod verify-offline --ignore-not-found # 6. 最终验证网络 kubectl run net-verify --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default # 预期DNS解析成功 # 7. 最终验证存储 kubectl get storageclass # 预期StorageClass存在 kubectl get pv | wc -l # 预期PV数量正常 # 8. 最终验证监控 kubectl get pods -n monitoring 2/dev/null # 预期监控Pod Running # 9. 生成操作报告 echo 离线环境操作报告 /tmp/offline_operation_report.txt echo 操作时间: $(date) /tmp/offline_operation_report.txt echo K8s版本: $(kubectl version --short 2/dev/null | grep Server) /tmp/offline_operation_report.txt echo 节点数: $(kubectl get nodes --no-headers | wc -l) /tmp/offline_operation_report.txt echo Pod数: $(kubectl get pods -A --no-headers | wc -l) /tmp/offline_operation_report.txt echo 节点状态: /tmp/offline_operation_report.txt kubectl get nodes /tmp/offline_operation_report.txt cat /tmp/offline_operation_report.txt # 10. 清理测试资源 docker rmi harbor.example.com/library/nginx:verify 2/dev/null echo 操作完成 验证流程# 1. K8s集群状态验证 kubectl get nodes -o wide # 预期所有节点 Ready kubectl get pods -A | grep -v Running | grep -v Completed # 预期无异常Pod # 2. 节点资源验证 kubectl top nodes # 预期CPU和内存使用率正常 kubectl top pods -A # 预期Pod资源使用正常 # 3. 网络连通性验证 kubectl run net-test --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default # 预期DNS解析成功 # 4. 容器运行时验证 systemctl status containerd # 预期active (running) ctr version # 预期版本正确 # 5. 离线镜像仓库验证 curl -k -I https://harbor.example.com # 预期返回200或302 # 6. 存储验证 kubectl get storageclass # 预期StorageClass存在 # 7. 监控验证 kubectl get pods -n monitoring 2/dev/null # 预期监控Pod Running # 8. 操作报告验证 cat /tmp/offline_operation_report.txt # 预期报告完整 排错方案K8s离线安装包缺失检查kubeadm/kubelet/kubectl、containerd、CNI插件、CoreDNS、metrics-server、镜像、依赖、版本、架构、完整性、校验kubeadm初始化失败检查系统要求、内核、依赖、网络、时间、主机名、swap、防火墙、SELinux、cgroup、容器运行时、镜像、配置、日志etcd集群部署失败检查节点数、网络、端口、证书、数据目录、磁盘、内存、配置、日志、健康检查、备份、恢复containerd离线安装失败检查版本、架构、依赖、配置、cgroup、镜像、注册表、证书、日志、测试CNI插件离线部署失败检查calico/flannel版本、镜像、配置、网段、网络策略、IPAM、日志、连通性测试CoreDNS离线部署失败检查镜像、配置、域名解析、上游DNS、日志、测试metrics-server离线安装失败检查镜像、证书、配置、API、指标、日志、测试节点join失败检查token、证书、网络、端口、版本、容器运行时、CNI、配置、日志、重试 K8S离线Ingress网关离线部署、内网反向代理闭环搭建
离线Ingress网关离线部署、内网反向代理闭环搭建技术栈Kubernetes v1.32.13 Rocky Linux 8.6 Containerd 1.7.x操作环境 / 对接原理 / 详细步骤 / 完整命令 / 配置文件 / 验证流程 / 排错方案离线Ingress网关离线部署、内网反向代理闭环搭建操作环境K8s集群3节点k8s-master(192.168.1.10), k8s-node1(192.168.1.11), k8s-node2(192.168.1.12)K8s版本 v1.32.13容器运行时 Containerd 1.7.x操作系统 Rocky Linux 8.6离线环境无外网物理隔离机房内网网段 192.168.1.0/24已搭建离线YUM源、离线镜像仓库(Harbor)、离线NTP服务离线场景K8s离线集群从零部署搭建已完成离线环境基础搭建K8s集群已部署离线镜像仓库已对接离线监控日志系统已部署存储配置本地存储/NFS/Ceph数据目录 /data已配置StorageClass和PVC动态供给已配置离线监控系统PrometheusGrafana可查看集群指标、节点指标、应用指标、日志、事件便于监控和排障对接原理离线Ingress网关离线部署、内网反向代理闭环搭建是 K8s 离线环境专属对接场景中的核心操作场景。离线环境是政企内网、涉密机房、私有化部署的典型形态无外网物理隔离带来了软件包获取、镜像拉取、漏洞库更新、时间同步、日志归集等一系列特殊挑战。K8s离线集群从零部署搭建作为离线环境运维的具体领域负责离线环境规划预检、K8s集群离线部署、离线镜像仓库运维、离线网络适配、离线存储对接、离线监控日志告警、离线CI/CD持续交付等核心能力。离线环境运维的核心目标是在无外网条件下实现K8s集群的稳定运行、安全可控、可追溯、可演进通过前置规划和环境预检确保离线环境基础达标通过离线安装包和依赖归档实现K8s集群从零部署通过离线镜像仓库和镜像全生命周期管理实现容器镜像的可控分发通过离线网络深度适配实现集群内外网络连通与隔离通过离线存储全场景适配实现数据持久化与安全通过离线监控日志告警体系实现无外网可观测通过离线CI/CD实现业务持续交付。所有操作需遵循先规划后实施、先备份后操作、先验证后上线的安全原则避免离线环境操作引发集群中断。详细步骤1. 离线环境现状盘点与基础状态检查# 1. 检查离线环境基础状态 ssh root192.168.1.10 # 检查系统信息 cat /etc/os-release uname -r # 检查网络 ip addr ip route ping -c 3 192.168.1.11 # 检查时间 timedatectl chronyc sources 2/dev/null || ntpq -p 2/dev/null # 检查主机名和hosts hostname cat /etc/hosts # 检查防火墙 firewall-cmd --state 2/dev/null firewall-cmd --list-ports 2/dev/null iptables -L -n | head -20 # 检查SELinux getenforce sestatus # 检查swap free -h swapon --show # 检查磁盘 df -h lsblk # 检查内核模块 lsmod | grep -E overlay|br_netfilter|ip_vs # 检查系统参数 sysctl net.ipv4.ip_forward sysctl net.bridge.bridge-nf-call-iptables sysctl net.bridge.bridge-nf-call-ip6tables # 2. 检查K8s集群状态 kubectl get nodes -o wide kubectl get pods -A -o wide kubectl get cs # 检查组件状态 kubectl get pods -n kube-system # 检查集群信息 kubectl cluster-info kubectl version # 3. 检查容器运行时 systemctl status containerd containerd --version ctr version # 检查containerd配置 cat /etc/containerd/config.toml | head -50 # 检查镜像 ctr images ls | head -20 # 4. 检查离线YUM源 cat /etc/yum.repos.d/*.repo yum repolist # 测试安装 yum list available | head -20 # 5. 检查离线镜像仓库 curl -k -I https://harbor.example.com 2/dev/null curl -k -u admin:${HARBOR_PASSWORD} https://harbor.example.com/api/v2.0/projects 2/dev/null | jq .[].name # 检查containerd仓库配置 cat /etc/containerd/config.toml | grep -A10 registry.mirrors # 6. 检查离线NTP chronyc tracking 2/dev/null chronyc sources -v 2/dev/null # 检查时间同步 timedatectl status # 7. 检查离线监控 curl -s http://prometheus:9090/api/v1/status/config 2/dev/null | head -20 curl -s http://grafana:3000/api/health 2/dev/null # 检查节点指标 kubectl top nodes kubectl top pods -A # 8. 导出当前配置备份 mkdir -p /tmp/offline_backup_$(date %Y%m%d) cp -r /etc/containerd /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null cp -r /etc/kubernetes /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null cp /etc/hosts /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null cp /etc/yum.repos.d/*.repo /tmp/offline_backup_$(date %Y%m%d)/ 2/dev/null echo 配置已备份到 /tmp/offline_backup_$(date %Y%m%d)/ 2. 制定操作方案与备份防护# 1. 制定离线环境操作方案 cat /tmp/offline_operation_plan.md EOF # 离线环境操作方案 ## 一、操作目标 ## 二、影响范围评估 ## 三、操作步骤与时间窗口 ## 四、回滚方案 ## 五、验证标准 ## 六、风险点与应对措施 ## 七、离线环境特殊注意事项 EOF echo 操作方案模板已创建 # 2. 确认离线资源可用性 # 检查离线安装包 ls -lh /opt/offline/packages/ 2/dev/null # 检查离线镜像 ls -lh /opt/offline/images/ 2/dev/null # 检查离线YUM源 df -h /var/www/html/yum/ 2/dev/null # 检查离线镜像仓库容量 df -h /data/harbor/ 2/dev/null # 3. 备份关键数据 # 备份K8s配置 kubectl get all -A -o yaml /tmp/k8s_all_backup_$(date %Y%m%d).yaml # 备份etcd kubectl -n kube-system exec etcd-k8s-master -- etcdctl snapshot save /tmp/etcd_snapshot_$(date %Y%m%d).db --endpointshttps://127.0.0.1:2379 --cacert/etc/kubernetes/pki/etcd/ca.crt --cert/etc/kubernetes/pki/etcd/server.crt --key/etc/kubernetes/pki/etcd/server.key 2/dev/null # 备份Harbor数据 cd /opt/harbor docker exec harbor-db pg_dump -U postgres registry /tmp/harbor_db_backup_$(date %Y%m%d).sql 2/dev/null # 备份配置文件 tar czf /tmp/offline_config_backup_$(date %Y%m%d).tar.gz /etc/kubernetes /etc/containerd /etc/hosts /etc/yum.repos.d 2/dev/null echo 关键数据已备份 # 4. 通知相关业务方 # echo 离线环境运维操作通知 | mail -s 离线环境通知 adminexample.com echo 建议在业务低峰期执行离线环境操作 # 5. 确认业务窗口 echo 当前时间: $(date) echo 离线环境操作需谨慎避免影响业务 3. 执行离线环境配置操作# 1. 检查离线环境配置文件 # 检查K8s配置 ls -la /etc/kubernetes/ cat /etc/kubernetes/admin.conf | head -20 # 检查kubelet配置 cat /var/lib/kubelet/config.yaml | head -30 # 检查kube-proxy配置 cat /var/lib/kube-proxy/config.conf 2/dev/null | head -30 # 2. 检查容器运行时配置 cat /etc/containerd/config.toml # 检查cgroup驱动 grep SystemdCgroup /etc/containerd/config.toml # 检查镜像仓库配置 grep -A20 registry /etc/containerd/config.toml # 3. 检查CNI配置 ls -la /etc/cni/net.d/ cat /etc/cni/net.d/10-calico.conflist 2/dev/null | head -30 cat /etc/cni/net.d/10-flannel.conflist 2/dev/null | head -30 # 检查Calico kubectl get pods -n kube-system | grep calico kubectl get ippools 2/dev/null # 4. 检查CoreDNS kubectl get pods -n kube-system | grep coredns kubectl get configmap coredns -n kube-system -o yaml | head -40 # 测试DNS kubectl run dns-test --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default 2/dev/null # 5. 检查存储配置 kubectl get storageclass kubectl get pv kubectl get pvc -A # 检查本地存储 ls -la /data/ # 检查NFS showmount -e localhost 2/dev/null mount | grep nfs # 6. 检查监控配置 kubectl get pods -n monitoring 2/dev/null kubectl get svc -n monitoring 2/dev/null # 检查Prometheus配置 kubectl get configmap prometheus-server -n monitoring -o yaml 2/dev/null | head -40 # 7. 执行具体离线环境配置操作根据标题调整 echo 执行离线环境具体配置操作... echo 请根据操作方案执行具体步骤 # 8. 应用配置变更 # 修改配置后重启相关服务 # systemctl restart kubelet # systemctl restart containerd # kubectl rollout restart deployment coredns -n kube-system # 9. 等待服务重启完成 sleep 30 kubectl get nodes kubectl get pods -A | grep -v Running | grep -v Completed echo 配置变更已应用 4. 验证集群网络存储镜像对接# 1. 验证K8s集群状态 kubectl get nodes -o wide # 预期所有节点 Ready kubectl get pods -A -o wide # 预期所有Pod Running或Completed # 2. 验证节点资源 kubectl top nodes # 预期CPU和内存使用率正常 kubectl top pods -A # 预期Pod资源使用正常 # 3. 验证网络连通性 # 测试Pod间通信 kubectl run net-test1 --imagebusybox --restartNever -n default -- sleep 3600 kubectl run net-test2 --imagebusybox --restartNever -n default -- sleep 3600 sleep 10 POD1_IP$(kubectl get pod net-test1 -o jsonpath{.status.podIP}) kubectl exec net-test2 -- ping -c 3 $POD1_IP # 预期ping通 kubectl delete pod net-test1 net-test2 --ignore-not-found # 4. 验证Service和DNS kubectl run dns-test --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default # 预期DNS解析成功 kubectl run dns-test2 --imagebusybox --restartNever --rm -it -- wget -q -O- https://kubernetes.default.svc.cluster.local 2/dev/null # 预期Service访问成功 # 5. 验证容器运行时 systemctl status containerd # 预期active (running) ctr version # 预期版本正确 ctr images ls | head -10 # 预期镜像列表正常 # 6. 验证离线镜像仓库 curl -k -I https://harbor.example.com # 预期返回200或302 # 测试镜像推拉 docker login harbor.example.com -u admin -p ${HARBOR_PASSWORD} docker pull nginx:latest docker tag nginx:latest harbor.example.com/library/nginx:test docker push harbor.example.com/library/nginx:test docker rmi harbor.example.com/library/nginx:test docker pull harbor.example.com/library/nginx:test # 预期推拉成功 # 7. 验证K8s从离线仓库拉取镜像 kubectl run test-harbor --imageharbor.example.com/library/nginx:test --restartNever sleep 10 kubectl get pods test-harbor # 预期Pod Running kubectl delete pod test-harbor --ignore-not-found # 8. 验证存储 # 创建测试PVC cat EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi EOF sleep 10 kubectl get pvc test-pvc # 预期Bound kubectl delete pvc test-pvc --ignore-not-found # 9. 验证监控 kubectl get pods -n monitoring 2/dev/null # 预期监控Pod Running # 检查指标 curl -s http://prometheus:9090/api/v1/query?queryup 2/dev/null | jq .data.result | length # 预期返回指标数量 # 10. 清理测试资源 docker rmi harbor.example.com/library/nginx:test 2/dev/null echo 验证完成 5. 离线YUM/NTP/镜像仓库/监控配置# 1. 配置离线YUM源 # 创建本地YUM源目录 mkdir -p /var/www/html/yum/rocky8 # 同步软件包在有网机器执行 # reposync -g -l -d -m --repoidbaseos --newest-only --download-metadata -p /var/www/html/yum/rocky8/ # 离线环境直接使用已同步的源 cat /etc/yum.repos.d/offline.repo EOF [offline-baseos] nameOffline BaseOS baseurlfile:///var/www/html/yum/rocky8/baseos enabled1 gpgcheck0 EOF yum clean all yum repolist # 2. 配置离线NTP # 安装chrony yum install -y chrony # 配置内网NTP服务器 cat /etc/chrony.conf EOF server 192.168.1.10 iburst allow 192.168.1.0/24 local stratum 10 EOF systemctl enable --now chronyd chronyc sources -v # 3. 配置离线镜像仓库 # Harbor已部署配置containerd对接 cat /etc/containerd/config.toml EOF version 2 [plugins.io.containerd.grpc.v1.cri.registry.mirrors.harbor.example.com] endpoint [https://harbor.example.com] [plugins.io.containerd.grpc.v1.cri.registry.configs.harbor.example.com.tls] ca_file /etc/containerd/certs.d/harbor.example.com/ca.crt EOF # 导入证书 mkdir -p /etc/containerd/certs.d/harbor.example.com cp /opt/harbor/ssl/ca.crt /etc/containerd/certs.d/harbor.example.com/ systemctl restart containerd # 4. 配置离线监控 # Prometheus配置 cat /etc/prometheus/prometheus.yml EOF global: scrape_interval: 15s scrape_configs: - job_name: kubernetes-nodes kubernetes_sd_configs: - role: node scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token EOF # 5. 配置离线日志 # 配置日志轮转 cat /etc/logrotate.d/k8s EOF /var/log/k8s/*.log { daily rotate 30 compress delaycompress missingok notifempty create 0644 root root } EOF # 6. 配置离线安全基线 # 关闭swap swapoff -a sed -i /swap/d /etc/fstab # 关闭SELinux setenforce 0 sed -i s/^SELINUX.*/SELINUXdisabled/ /etc/selinux/config # 配置内核参数 cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl --system # 7. 配置离线CI/CD # Jenkins配置 # 管理Jenkins插件离线安装 # 配置Jenkins任务对接离线镜像仓库 echo 离线CI/CD配置完成 6. 最终验证与操作报告# 1. 最终验证K8s集群状态 kubectl get nodes -o wide # 预期所有节点 Ready kubectl get pods -A | grep -v Running | grep -v Completed # 预期无异常Pod # 2. 最终验证离线YUM源 yum repolist # 预期离线源可用 yum list available | wc -l # 预期返回软件包数量 # 3. 最终验证离线NTP chronyc tracking # 预期时间同步正常 timedatectl status # 预期时间正确 # 4. 最终验证离线镜像仓库 curl -k -I https://harbor.example.com # 预期返回200或302 docker login harbor.example.com -u admin -p ${HARBOR_PASSWORD} docker pull nginx:latest docker tag nginx:latest harbor.example.com/library/nginx:verify docker push harbor.example.com/library/nginx:verify docker rmi harbor.example.com/library/nginx:verify docker pull harbor.example.com/library/nginx:verify # 预期推拉成功 # 5. 最终验证K8s对接离线仓库 kubectl run verify-offline --imageharbor.example.com/library/nginx:verify --restartNever sleep 10 kubectl get pods verify-offline # 预期Pod Running kubectl delete pod verify-offline --ignore-not-found # 6. 最终验证网络 kubectl run net-verify --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default # 预期DNS解析成功 # 7. 最终验证存储 kubectl get storageclass # 预期StorageClass存在 kubectl get pv | wc -l # 预期PV数量正常 # 8. 最终验证监控 kubectl get pods -n monitoring 2/dev/null # 预期监控Pod Running # 9. 生成操作报告 echo 离线环境操作报告 /tmp/offline_operation_report.txt echo 操作时间: $(date) /tmp/offline_operation_report.txt echo K8s版本: $(kubectl version --short 2/dev/null | grep Server) /tmp/offline_operation_report.txt echo 节点数: $(kubectl get nodes --no-headers | wc -l) /tmp/offline_operation_report.txt echo Pod数: $(kubectl get pods -A --no-headers | wc -l) /tmp/offline_operation_report.txt echo 节点状态: /tmp/offline_operation_report.txt kubectl get nodes /tmp/offline_operation_report.txt cat /tmp/offline_operation_report.txt # 10. 清理测试资源 docker rmi harbor.example.com/library/nginx:verify 2/dev/null echo 操作完成 验证流程# 1. K8s集群状态验证 kubectl get nodes -o wide # 预期所有节点 Ready kubectl get pods -A | grep -v Running | grep -v Completed # 预期无异常Pod # 2. 节点资源验证 kubectl top nodes # 预期CPU和内存使用率正常 kubectl top pods -A # 预期Pod资源使用正常 # 3. 网络连通性验证 kubectl run net-test --imagebusybox --restartNever --rm -it -- nslookup kubernetes.default # 预期DNS解析成功 # 4. 容器运行时验证 systemctl status containerd # 预期active (running) ctr version # 预期版本正确 # 5. 离线镜像仓库验证 curl -k -I https://harbor.example.com # 预期返回200或302 # 6. 存储验证 kubectl get storageclass # 预期StorageClass存在 # 7. 监控验证 kubectl get pods -n monitoring 2/dev/null # 预期监控Pod Running # 8. 操作报告验证 cat /tmp/offline_operation_report.txt # 预期报告完整 排错方案K8s离线安装包缺失检查kubeadm/kubelet/kubectl、containerd、CNI插件、CoreDNS、metrics-server、镜像、依赖、版本、架构、完整性、校验kubeadm初始化失败检查系统要求、内核、依赖、网络、时间、主机名、swap、防火墙、SELinux、cgroup、容器运行时、镜像、配置、日志etcd集群部署失败检查节点数、网络、端口、证书、数据目录、磁盘、内存、配置、日志、健康检查、备份、恢复containerd离线安装失败检查版本、架构、依赖、配置、cgroup、镜像、注册表、证书、日志、测试CNI插件离线部署失败检查calico/flannel版本、镜像、配置、网段、网络策略、IPAM、日志、连通性测试CoreDNS离线部署失败检查镜像、配置、域名解析、上游DNS、日志、测试metrics-server离线安装失败检查镜像、证书、配置、API、指标、日志、测试节点join失败检查token、证书、网络、端口、版本、容器运行时、CNI、配置、日志、重试