KubeEdge keadm debug 实战指南边缘节点的诊断、环境检查与故障数据收集【免费下载链接】kubeedgeKubernetes Native Edge Computing Framework (project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/ku/kubeedgeKubeEdge 的边缘节点Edge Node与云侧完全解耦运行传统上依赖kubectl的排障手段在边缘侧并不适用。KubeEdge 为此在 keadm 工具中提供了keadm debug命令组设计提案覆盖get、check、collect、diagnose四个子命令帮助运维人员无需连到云侧即可定位边缘节点故障。读完本文你可以掌握如何在边缘节点上直接查询本地数据库中的 Pod/Service 等资源、如何按阈值检查节点是否满足 edgecore 运行条件、如何一键打包节点全量诊断数据以及各命令背后的源码实现原理。背景与设计目标提案的动机是KubeEdge 边缘节点此前缺少好用的调试与诊断手段可能阻碍用户尝试 KubeEdge。因此目标被分为两个阶段Alpha收集当前环境与 kubeedge 相关的全部信息并以格式化方式提供给运维人员用于定位和解决疑难问题Beta全方位诊断特定故障场景并定位故障原因检查系统特定项是否满足 edgecore 安装与运行的要求。由此确定了首批基础命令集所有命令均建议在 root 用户下执行非 root 时请加sudokeadm debug getkeadm debug checkkeadm debug collectkeadm debug diagnose四个子命令均在 debug.go 的NewEdgeDebug()中注册整体帮助输出为keadm debug command help provide debug function to help diagnose the cluster Usage: keadm debug [command] Available Commands: check Check specific information. collect Obtain all the data of the current node diagnose Diagnose relevant information at edge nodes get Display one or many resources Flags: -h, --help help for debug Use keadm debug [command] --help for more information about a command.keadm debug get直接查询边缘节点本地数据库功能与用法keadm debug get从边缘节点本地数据库默认/var/lib/kubeedge/edgecore.db即 edgecore 的 metamanager SQLite 库读取资源信息并格式化输出。它的作用等价于“离线版 kubectl get”——即使节点与云侧断开也能查看云侧下发到边缘的资源副本。Usage: keadm debug get [flags] Examples: # List all pod in namespace test keadm debug get pod -n test # List a single configmap with specified NAME keadm debug get configmap web -n default # List the complete information of the configmap with the specified name in the yaml output format keadm debug get configmap web -n default -o yaml # List the complete information of all available resources of edge nodes using the specified format (default: yaml) keadm debug get all -o yaml Flags: -A, --all-namespaces List the requested object(s) across all namespaces -p, --edgedb-path string Indicate the edge node database path, the default path is /var/lib/kubeedge/edgecore.db -h, --help help for get -n, --namespace string List the requested object(s) in specified namespaces (default default) -o, --output string Indicate the output format. Currently supports formats such as yaml|json|wide -l, --selector string Selector (label query) to filter on, supports , , and !.(e.g. -l key1value1,key2value2)支持的资源类型与输出从源码 get.go 中的availableResources映射看当前支持的资源类型及其简写为资源类型可接受的写法Podpod、po、podsNodenode、no、nodesServiceservice、svc、servicesSecretsecret、secretsConfigMapconfigmap、cm、configmapsEndpointsendpoint、ep、endpoints全部all注意all后不能再跟资源名-o支持yaml、json以及默认/wide表格输出。实际运行示例提案原文示例列出 Pod[rootlocalhost bin]# keadm debug get pod NAME READY STATUS RESTARTS AGE nginx-ds-85jch 1/1 Running 4 21d nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 98 5d21h显式指定数据库文件路径-p[rootlocalhost tmp]# keadm debug get pod -p /var/lib/kubeedge/edgecore.db NAME READY STATUS RESTARTS AGE nginx-ds-85jch 1/1 Running 5 21d nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 105 5d22hwide 表格输出[rootlocalhost bin]# keadm debug get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ds-85jch 1/1 Running 4 21d 172.17.0.3 centos-kubeedge.shared none none nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 98 5d21h 172.17.0.4 centos-kubeedge.shared none none单个 Pod 的 json 输出[rootlocalhost bin]# keadm debug get pod nginx-ds-85jch -o json { apiVersion: v1, kind: pod, metadata: { creationTimestamp: 2020-10-19T02:55:07Z, generateName: nginx-ds-, labels: { controller-revision-hash: 69b66994b8, name: nginx-ds, pod-template-generation: 1 }, ... }, spec: { ... }, status: { ... } }跨所有命名空间查询[rootlocalhost bin]# keadm debug get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-9jh2l 0/1 Running 4 21d kube-system kube-proxy-2qrdt 1/1 Running 0 73d default nginx-ds-85jch 1/1 Running 4 21d default nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 98 5d22hConfigMap 与 Secret[rootlocalhost tmp]# keadm debug get cm -A NAMESPACE NAME DATA AGE calico-system typha-ca 1 118d kube-system kube-proxy 2 118d calico-system cni-config 1 118d kube-system calico-config 4 116d[rootlocalhost tmp]# keadm debug get secret -A NAMESPACE NAME TYPE DATA AGE kubeedge default-token-44mgz kubernetes.io/service-account-token 3 118d calico-system calico-node-token-q28zz kubernetes.io/service-account-token 3 118d calico-system calico-typha-token-2jnz2 kubernetes.io/service-account-token 3 118d calico-system calico-typha-token-xv4qk kubernetes.io/service-account-token 3 116d calico-system calico-node-token-8kjcp kubernetes.io/service-account-token 3 116d calico-system calico-kube-controllers-token-hkz7h kubernetes.io/service-account-token 3 116d kube-system calico-kube-controllers-token-v4kfx kubernetes.io/service-account-token 3 116d calico-system node-certs Opaque 3 116d calico-system typha-certs Opaque 3 116d kube-system kube-proxy-token-nqnvx kubernetes.io/service-account-token 3 118d kube-system calico-node-token-lfmzn kubernetes.io/service-account-token 3 116d default default-token-w5skc kubernetes.io/service-account-token 3 118dEndpoints[rootlocalhost tmp]# keadm debug get ep NAME ENDPOINTS AGE kubernetes 10.211.55.6:6443 118d nginx 192.168.1.209:80,192.168.1.235:80,192.168.1.237:80 73d源码实现解析get的执行链路get.go为Validate→Run→queryDataFromDatabase→ 打印。几个值得注意的实现细节数据库访问方式InitDBget.go#L458-L481使用 GORM 的 sqlite 驱动打开edgecore.db并复用 edge/pkg/metamanager 的dbclient.NewMetaService()查询models.Meta表——这与 edgecore 内部 metamanager 持久化资源的方式完全一致因此读到的就是边缘侧“当前视图”。Pod 状态的拼接getPodsFromDatabaseget.go#L297-L341会先查询pod类型记录再按 key 规则把pod替换为podstatus查出对应的PodStatusRequest将其中Status字段合并进 Pod 的status后再输出。这解释了为什么表格中STATUS/READY列有值——它们来自独立的 podstatus 记录而非 Pod 对象本身。输出渲染表格输出复用了 kubectl 的printers/TableConvertor体系get.go#L642-L648 的ConvertDataToTable列定义与 api-server 的 Table 输出保持一致-o json|yaml则走JSONYamlPrint将 Meta 还原为标准v1.Pod/v1.Service等对象后序列化。get_flags.go 的文件头注释也说明了这一点该文件衍生自 k8s kubectl 的get_flags.go并做了精简。标签过滤-l选择器由SplitSelectorParameters解析支持、与!三种比较get.go#L502-L542过滤发生在数据库查询结果之上基于对象metadata.labels。keadm debug check检查节点是否满足 edgecore 运行条件功能与用法check用于验证系统特定项是否满足 edgecore 安装与运行的要求覆盖 CPU、内存、磁盘、DNS、网络、PID 限制和容器运行时七类检查项可用all一次执行全部Usage: keadm debug check [command] Examples: # Check all items . keadm debug check all # Check whether the node CPU meets requirements. keadm debug check cpu # Check whether the node memory meets requirements. keadm debug check mem # check whether the node disk meets requirements. keadm debug check disk # Check whether the node DNS can resolve a specific domain name. keadm debug check dns -d www.github.com # Check whether the node network meets requirements. keadm debug check network # Check whether the number of free processes on the node meets requirements. keadm debug check pid # Check whether runtime(Docker) is installed on the node. keadm debug check runtime Available Commands: all Check all item cpu Check node CPU requirements disk Check node disk requirements dns Check whether DNS can work mem Check node memory requirements network Check whether the network is normal pid Check node PID requirements runtime Check whether runtime can work各检查项的判定阈值从源码 check.go 与阈值常量 constant.go#L219-L222 可以确认每项的判定标准子命令判定条件关键参数cpu核心数 ≥ 1 vCore且 CPU 使用率 90%无mem内存总量 256MB空闲内存 128MB使用率 90%无disk磁盘总量 1GB空闲 512MB使用率 90%无dns域名可解析默认测试域名为www.github.com-d指定域名-D指定 DNS 服务器 IPnetworkping 测试 IP 零丢包HTTPS 连通 cloudhubHTTP 连通本地127.0.0.1:10350-i指定测试 IP缺省自动取/etc/resolv.conf中的 nameserver-s指定 cloudhub 地址缺省从 edgecore 配置读取-c指定配置文件pid空闲进程比例 (1 - 当前进程数/最大 PID 数) 5%无runtime容器运行时可连接且处于运行状态-c指定 edgecore 配置文件从源码实现看几个细节check dns指定 DNS 服务器CheckDNSSpecify 通过替换net.DefaultResolver的 Dial 函数强制向指定IP:53的 UDP 端口发起解析从而验证“某个 DNS 服务器能否解析某个域名”——这在验证集群内 kube-dnsCoreDNS时非常有用。check network的三级验证CheckNetWork 依次执行 ①ping测试 IP未指定时通过 CmdGetDNSIP 从/etc/resolv.conf提取 nameserver② 对 edgecore 配置中edgehub.webSocket.server发起 HTTPS 请求CheckHTTP 对 x509 证书错误视为可达因为网络层已通③ 对127.0.0.1:10350EdgeCoreServer 常量发起 HTTP 请求验证本机 edgecore 服务存活。check runtime的运行时来源提案中曾设想用-r参数指定容器运行时从当前源码看getRuntimeEndpointcheck.go#L375-L402改为优先从 edgecore 配置modules.edged.tailoredKubeletConfig.containerRuntimeEndpoint读取运行时端点未配置时回退到默认 remote runtime 端点随后复用 kubeadm 的utilruntime.NewContainerRuntime完成 Connect IsRunning 两步检查。check pidCheckPid 通过 shell 读取系统最大进程数与当前进程数空闲比例不足 5% 即判定失败这与提案中“可用进程数少于 5% 视为不足”的描述一致。硬件指标采集统一使用gopsutilcpu/mem/disk三个包不依赖特定发行版的命令输出格式。实际运行示例提案原文示例[rootlocalhost tmp]# keadm debug check cpu CPU total: 1 core, Allowed 1 core CPU usage rate: 0.00, Allowed rate 0.9 |-----------------| ||check cpu succeed| |-----------------|[rootlocalhost tmp]# keadm debug check mem Memory total: 1833.33 MB, Allowed 256 MB Memory Free total: 1074.40 MB, Allowed 128 MB Memory usage rate: 0.11, Allowed rate 0.9 |-----------------| ||check mem succeed| |-----------------|[rootlocalhost tmp]# keadm debug check disk Disk total: 50268.47 MB, Allowed 1024 MB Disk Free total: 39087.80 MB, Allowed 512MB Disk usage rate: 0.18, Allowed rate 0.9 |------------------| ||check disk succeed| |------------------|[rootlocalhost tmp]# keadm debug check dns dns resolution success, domain: www.github.com ip: 192.30.255.113 |-----------------| ||check dns succeed| |-----------------|验证集群内 K8s Service DNS 解析先取得 kube-dns 的 ClusterIP 作为 DNS 服务器[rootk8s ~]# kubectl get svc -n kube-system |grep dns kube-dns ClusterIP 10.96.0.10 none 53/UDP,53/TCP,9153/TCP 118d [rootk8s ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 none 443/TCP 118d nginx ClusterIP 10.97.180.216 none 80/TCP 73d [rootlocalhost tmp]# keadm debug check dns -D 10.96.0.10 -d nginx dns resolution success, domain: nginx ip: 10.97.180.216 |-----------------| ||check dns succeed| |-----------------|[rootlocalhost tmp]# keadm debug check network ping 172.17.0.1 success check cloudhubServer 10.211.55.6:10000 success check edgecoreServer 127.0.0.1:10350 success |---------------------| ||check network succeed| |---------------------|Maximum PIDs: 32768; Running processes: 129 |-----------------| ||check pid succeed| |-----------------|[rootlocalhost tmp]# keadm debug check runtime docker is running |---------------------| ||check runtime succeed| |---------------------|check all汇总输出[rootlocalhost tmp]# keadm debug check all CPU total: 1 core, Allowed 1 core CPU usage rate: 0.10, Allowed rate 0.9 Memory total: 1833.33 MB, Allowed 256 MB Memory Free total: 605.50 MB, Allowed 128 MB Memory usage rate: 0.17, Allowed rate 0.9 Disk total: 50268.47 MB, Allowed 1024 MB Disk Free total: 39085.29 MB, Allowed 512MB Disk usage rate: 0.18, Allowed rate 0.9 dns resolution success, domain: www.github.com ip: 192.30.255.112 ping 172.17.0.1 success check cloudhubServer 10.211.55.6:10000 success check edgecoreServer 127.0.0.1:10350 success Maximum PIDs: 32768; Running processes: 129 docker is running |-----------------| ||check all succeed| |-----------------|keadm debug collect一键打包节点全量诊断数据功能与用法collect将当前节点上与 KubeEdge 排查相关的全部数据收集、压缩为edge-时间戳.tar.gz供运维人员离线分析Usage: keadm debug collect [flags] Examples: keadm debug collect --path . # Collect all items and specified the output directory path keadm debug collect --output-path . Flags: -c, --config string Specify configuration file, default is /etc/kubeedge/config/edgecore.yaml -d, --detail Whether to print internal log output -h, --help help for collect -l, --log-path string Specify log file (default /var/log/kubeedge/) -o, --output-path string Cache data and store data compression packages in a directory that default to the current directory参数说明与 collect.go 中的 flag 定义一致-c, --configedgecore 配置文件默认/etc/kubeedge/config/edgecore.yaml。源码中VerificationParameters会校验该文件存在不存在直接报错-d, --detail打印每一步的内部执行日志创建临时目录、每条 shell 命令、每个文件复制-l, --log-path日志目录默认/var/log/kubeedge/-o, --output-path压缩包的输出目录默认当前目录.。注意源码会将其解析为绝对路径并要求该目录已存在否则会报output-path ... does not exist。收集范围按提案与源码 collect.go数据分为三大类1. 系统数据collectSystemDatacollect.go#L157-L206数据项采集方式硬件架构archCPU 信息复制/proc/cpuinfo内存信息复制/proc/meminfo磁盘信息df -h网络信息netstat -anp复制/etc/resolv.conf与/etc/hosts进程信息ps -aux时间信息date与uptime历史命令history -a cat ~/.bash_history2. Edgecore 数据collectEdgecoreDatacollect.go#L209-L268数据库文件优先取配置中dataBase.dataSource缺省为/var/lib/kubeedge/edgecore.db日志-l指定的目录或默认/var/log/kubeedge/systemd 服务文件/lib/systemd/system/edgecore.service配置文件目录/etc/kubeedge/config/证书优先取配置中modules.edgeHub.tlsCertFile/tlsPrivateKeyFile与tlsCaFile缺省回退到/etc/kubeedge/certs/与默认 CA 路径软件版本执行edgecore --version。3. 容器运行时数据源码中定义了collectRuntimeDatacollect.go#L271-L290用于收集docker version、docker info、docker images、docker ps -a、runtime 日志及docker.service等。从当前源码结构看ExecuteCollect流程中该步骤尚带// TODO: collectRuntimeData with containerd的适配注释因此不同版本中 runtime 部分可能未出现在压缩包里使用时以实际输出为准。实际运行示例提案原文示例默认收集[rootlocalhost tmp]# keadm debug collect Start collecting data Data collected successfully, path: /root/tmp/edge_2020_1109_173744.tar.gz-d打开明细可看到完整的采集过程临时目录位于/tmp/edge_时间戳按 system/edgecore/runtime 分子目录组织[rootlocalhost tmp]# keadm debug collect -d Start collecting data create tmp file: /tmp/edge_2020_1109_173800 create tmp file: /tmp/edge_2020_1109_173800/system Execute Shell: arch /tmp/edge_2020_1109_173800/system/arch Copy File: cp -r /proc/cpuinfo /tmp/edge_2020_1109_173800/system/ Copy File: cp -r /proc/meminfo /tmp/edge_2020_1109_173800/system/ Execute Shell: df -h /tmp/edge_2020_1109_173800/system/disk Copy File: cp -r /etc/hosts /tmp/edge_2020_1109_173800/system/ Copy File: cp -r /etc/resolv.conf /tmp/edge_2020_1109_173800/system/ Execute Shell: ps -axu /tmp/edge_2020_1109_173800/system/process Execute Shell: date /tmp/edge_2020_1109_173800/system/date Execute Shell: uptime /tmp/edge_2020_1109_173800/system/uptime Execute Shell: history -a cat ~/.bash_history /tmp/edge_2020_1109_173800/system/history Execute Shell: netstat -pan /tmp/edge_2020_1109_173800/system/network collect systemd data finish create tmp file: /tmp/edge_2020_1109_173800/edgecore Copy File: cp -r /var/lib/kubeedge/edgecore.db /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /var/log/kubeedge/ /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /lib/systemd/system/edgecore.service /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/config/ /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/certs/server.crt /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/certs/server.key /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/ca/rootCA.crt /tmp/edge_2020_1109_173800/edgecore/ Execute Shell: edgecore --version /tmp/edge_2020_1109_173800/edgecore/version collect edgecore data finish create tmp file: /tmp/edge_2020_1109_173800/runtime Copy File: cp -r /lib/systemd/system/docker.service /tmp/edge_2020_1109_173800/runtime/ Execute Shell: docker version /tmp/edge_2020_1109_173800/runtime/version Execute Shell: docker info /tmp/edge_2020_1109_173800/runtime/info Execute Shell: docker images /tmp/edge_2020_1109_173800/runtime/images Execute Shell: docker ps -a /tmp/edge_2020_1109_173800/runtime/containerInfo Execute Shell: journalctl -u docker /tmp/edge_2020_1109_173800/runtime/log collect runtime data finish Data compressed successfully Remove tmp data finish Data collected successfully, path: /root/tmp/edge_2020_1109_173800.tar.gz指定输出目录与配置文件[rootlocalhost tmp]# keadm debug collect -o /tmp Start collecting data Data collected successfully, path: /tmp/edge_2020_1109_173842.tar.gz[rootlocalhost tmp]# keadm debug collect -c /etc/kubeedge/config/edgecore.yaml Start collecting data Data collected successfully, path: /root/tmp/edge_2020_1109_173931.tar.gz整体流程ExecuteCollect为参数校验 → 在/tmp下创建edge_时间戳临时目录 → 采集系统数据 → 解析 edgecore 配置并采集 edgecore 数据 →util.Compress打包为edge_时间戳.tar.gz→ 清理临时目录。注意压缩包中会包含证书文件提交给他人排查时请知悉其中的敏感信息。keadm debug diagnose面向故障场景的诊断功能与用法diagnose针对特定故障场景做“结论式”诊断直接输出成功/失败Usage: keadm debug diagnose [command] Examples: # Diagnose whether the node is normal keadm debug diagnose node # Diagnose whether the pod is normal keadm debug diagnose pod nginx-xxx -n test # Diagnose node installation conditions keadm debug diagnose install # Diagnose node installation conditions and specify the detected ip keadm debug diagnose install -i 192.168.1.2 Available Commands: install Diagnose install node Diagnose edge node pod Diagnose pod各子命令的诊断范围与源码行为diagnose nodeDiagnoseNode依次检查edgecore 进程是否正在运行edgecore 配置文件是否存在默认/etc/kubeedge/config/edgecore.yaml可用-c覆盖数据库文件是否存在配置dataBase.dataSource缺省/var/lib/kubeedge/edgecore.dbedgehub websocket 是否启用并对配置中的 cloudhub 地址发起 HTTPS 连通性检查。diagnose pod name [-n ns]源码中diagnose pod会先执行完整的 node 诊断diagnose.go#L96-L105通过后打开本地数据库按 key 规则查询ns/pod/name与ns/podstatus/name两条记录检查 Pod phase、ConditionsReady 等与容器状态Waiting/Terminated 的 message、reason、RestartCount最终给出Pod xxx is Ready或失败原因。diagnose install与keadm debug check all语义相近DiagnoseInstall依次执行 CPU、内存、磁盘、DNS、网络、PID 检查用于安装前/重装前验证节点条件。参数-i指定测试 IP、-d指定测试域名、-D指定 DNS 服务器 IP、-s指定 cloudhub 地址。实际运行示例提案原文示例节点健康[rootlocalhost tmp]# keadm debug diagnose node edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection success |---------------------| ||diagnose node succeed| |---------------------|Pod 诊断成功[rootlocalhost tmp]# keadm debug diagnose pod nginx-ds-85jch -n default edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection successDatabase /var/lib/kubeedge/edgecore.db is exist Pod nginx-ds-85jch is exist PodStatus nginx-ds-85jch is exist pod nginx-ds-85jch phase is Running containerConditions nginx-ds is ready Pod nginx-ds-85jch is Ready |--------------------| ||diagnose pod succeed| |--------------------|Pod 在指定 namespace 下不存在[rootlocalhost tmp]# keadm debug diagnose pod nginx-ds-85jch -n kube-system edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection successDatabase /var/lib/kubeedge/edgecore.db is exist not find kube-system/pod/nginx-ds-85jch in database |-------------------| ||diagnose pod failed| |-------------------|Pod 存在但容器无法启动典型的故障定位输出直接给出 OCI 报错原因与重启次数[rootlocalhost tmp]# keadm debug diagnose pod nginx-deployment-dbbffc676-wprs8 edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection successDatabase /var/lib/kubeedge/edgecore.db is exist Pod nginx-deployment-dbbffc676-wprs8 is exist PodStatus nginx-deployment-dbbffc676-wprs8 is exist pod nginx-deployment-dbbffc676-wprs8 phase is Running conditions is not true, type: Ready ,message: containers with unready status: [nginx] ,reason: ContainersNotReady containerConditions nginx Terminated, message: oci runtime error: container_linux.go:235: starting container process caused exec: \/abc\: stat /abc: no such file or directory , reason: ContainerCannotRun, RestartCount: 104 Pod nginx-deployment-dbbffc676-wprs8 is not Ready |-------------------| ||diagnose pod failed| |-------------------|可以看到诊断输出直接定位到“容器 entrypoint/abc不存在”这一根因这正是提案希望达成的“定位故障原因”效果。推荐的排障工作流结合四个命令的定位建议按以下顺序在边缘节点上排障快速健康检查keadm debug diagnose node——确认 edgecore 进程、配置、数据库、到 cloudhub 的连通性四项基本盘环境与安装条件keadm debug check all——确认资源、DNS、网络、PID、运行时是否满足 edgecore 运行要求部署新节点时用keadm debug diagnose install做前置验证资源层面核对keadm debug get pod -A/keadm debug get all -o yaml——确认云侧下发的资源是否已同步到本地数据库Pod 状态停在什么阶段单点深入keadm debug diagnose pod name -n ns——拿到 Ready 条件与容器级报错留档求助keadm debug collect -d——打包节点全量数据交给进一步分析。适用前提与注意事项命令需在边缘节点本机执行且建议以 root或加sudo运行因为 collect 需要读取系统文件与证书目录get/diagnose的数据源是 edgecore 本地数据库默认路径/var/lib/kubeedge/edgecore.db可用-p覆盖若 edgecore 配置中dataBase.dataSource指向其他路径diagnose 会以配置为准check network与diagnose node的 cloudhub 地址默认从 edgecore 配置默认/etc/kubeedge/config/edgecore.yaml中的modules.edgeHub.webSocket.server读取也可用-s显式覆盖本文示例输出取自提案文档 keadm-debug.md 中的实测记录不同 KubeEdge 版本的实际命令输出格式可能略有差异参数与判定逻辑以当前仓库 keadm/cmd/keadm/app/cmd/debug/ 下的源码为准单元测试可参考 check_test.go、collect_test.go、diagnose_test.go 与 get_flags_test.go。【免费下载链接】kubeedgeKubernetes Native Edge Computing Framework (project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/ku/kubeedge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 KubeEdge keadm debug 实战指南：边缘节点的诊断、环境检查与故障数据收集
KubeEdge keadm debug 实战指南边缘节点的诊断、环境检查与故障数据收集【免费下载链接】kubeedgeKubernetes Native Edge Computing Framework (project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/ku/kubeedgeKubeEdge 的边缘节点Edge Node与云侧完全解耦运行传统上依赖kubectl的排障手段在边缘侧并不适用。KubeEdge 为此在 keadm 工具中提供了keadm debug命令组设计提案覆盖get、check、collect、diagnose四个子命令帮助运维人员无需连到云侧即可定位边缘节点故障。读完本文你可以掌握如何在边缘节点上直接查询本地数据库中的 Pod/Service 等资源、如何按阈值检查节点是否满足 edgecore 运行条件、如何一键打包节点全量诊断数据以及各命令背后的源码实现原理。背景与设计目标提案的动机是KubeEdge 边缘节点此前缺少好用的调试与诊断手段可能阻碍用户尝试 KubeEdge。因此目标被分为两个阶段Alpha收集当前环境与 kubeedge 相关的全部信息并以格式化方式提供给运维人员用于定位和解决疑难问题Beta全方位诊断特定故障场景并定位故障原因检查系统特定项是否满足 edgecore 安装与运行的要求。由此确定了首批基础命令集所有命令均建议在 root 用户下执行非 root 时请加sudokeadm debug getkeadm debug checkkeadm debug collectkeadm debug diagnose四个子命令均在 debug.go 的NewEdgeDebug()中注册整体帮助输出为keadm debug command help provide debug function to help diagnose the cluster Usage: keadm debug [command] Available Commands: check Check specific information. collect Obtain all the data of the current node diagnose Diagnose relevant information at edge nodes get Display one or many resources Flags: -h, --help help for debug Use keadm debug [command] --help for more information about a command.keadm debug get直接查询边缘节点本地数据库功能与用法keadm debug get从边缘节点本地数据库默认/var/lib/kubeedge/edgecore.db即 edgecore 的 metamanager SQLite 库读取资源信息并格式化输出。它的作用等价于“离线版 kubectl get”——即使节点与云侧断开也能查看云侧下发到边缘的资源副本。Usage: keadm debug get [flags] Examples: # List all pod in namespace test keadm debug get pod -n test # List a single configmap with specified NAME keadm debug get configmap web -n default # List the complete information of the configmap with the specified name in the yaml output format keadm debug get configmap web -n default -o yaml # List the complete information of all available resources of edge nodes using the specified format (default: yaml) keadm debug get all -o yaml Flags: -A, --all-namespaces List the requested object(s) across all namespaces -p, --edgedb-path string Indicate the edge node database path, the default path is /var/lib/kubeedge/edgecore.db -h, --help help for get -n, --namespace string List the requested object(s) in specified namespaces (default default) -o, --output string Indicate the output format. Currently supports formats such as yaml|json|wide -l, --selector string Selector (label query) to filter on, supports , , and !.(e.g. -l key1value1,key2value2)支持的资源类型与输出从源码 get.go 中的availableResources映射看当前支持的资源类型及其简写为资源类型可接受的写法Podpod、po、podsNodenode、no、nodesServiceservice、svc、servicesSecretsecret、secretsConfigMapconfigmap、cm、configmapsEndpointsendpoint、ep、endpoints全部all注意all后不能再跟资源名-o支持yaml、json以及默认/wide表格输出。实际运行示例提案原文示例列出 Pod[rootlocalhost bin]# keadm debug get pod NAME READY STATUS RESTARTS AGE nginx-ds-85jch 1/1 Running 4 21d nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 98 5d21h显式指定数据库文件路径-p[rootlocalhost tmp]# keadm debug get pod -p /var/lib/kubeedge/edgecore.db NAME READY STATUS RESTARTS AGE nginx-ds-85jch 1/1 Running 5 21d nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 105 5d22hwide 表格输出[rootlocalhost bin]# keadm debug get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ds-85jch 1/1 Running 4 21d 172.17.0.3 centos-kubeedge.shared none none nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 98 5d21h 172.17.0.4 centos-kubeedge.shared none none单个 Pod 的 json 输出[rootlocalhost bin]# keadm debug get pod nginx-ds-85jch -o json { apiVersion: v1, kind: pod, metadata: { creationTimestamp: 2020-10-19T02:55:07Z, generateName: nginx-ds-, labels: { controller-revision-hash: 69b66994b8, name: nginx-ds, pod-template-generation: 1 }, ... }, spec: { ... }, status: { ... } }跨所有命名空间查询[rootlocalhost bin]# keadm debug get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-9jh2l 0/1 Running 4 21d kube-system kube-proxy-2qrdt 1/1 Running 0 73d default nginx-ds-85jch 1/1 Running 4 21d default nginx-deployment-dbbffc676-wprs8 0/1 ContainerCannotRun 98 5d22hConfigMap 与 Secret[rootlocalhost tmp]# keadm debug get cm -A NAMESPACE NAME DATA AGE calico-system typha-ca 1 118d kube-system kube-proxy 2 118d calico-system cni-config 1 118d kube-system calico-config 4 116d[rootlocalhost tmp]# keadm debug get secret -A NAMESPACE NAME TYPE DATA AGE kubeedge default-token-44mgz kubernetes.io/service-account-token 3 118d calico-system calico-node-token-q28zz kubernetes.io/service-account-token 3 118d calico-system calico-typha-token-2jnz2 kubernetes.io/service-account-token 3 118d calico-system calico-typha-token-xv4qk kubernetes.io/service-account-token 3 116d calico-system calico-node-token-8kjcp kubernetes.io/service-account-token 3 116d calico-system calico-kube-controllers-token-hkz7h kubernetes.io/service-account-token 3 116d kube-system calico-kube-controllers-token-v4kfx kubernetes.io/service-account-token 3 116d calico-system node-certs Opaque 3 116d calico-system typha-certs Opaque 3 116d kube-system kube-proxy-token-nqnvx kubernetes.io/service-account-token 3 118d kube-system calico-node-token-lfmzn kubernetes.io/service-account-token 3 116d default default-token-w5skc kubernetes.io/service-account-token 3 118dEndpoints[rootlocalhost tmp]# keadm debug get ep NAME ENDPOINTS AGE kubernetes 10.211.55.6:6443 118d nginx 192.168.1.209:80,192.168.1.235:80,192.168.1.237:80 73d源码实现解析get的执行链路get.go为Validate→Run→queryDataFromDatabase→ 打印。几个值得注意的实现细节数据库访问方式InitDBget.go#L458-L481使用 GORM 的 sqlite 驱动打开edgecore.db并复用 edge/pkg/metamanager 的dbclient.NewMetaService()查询models.Meta表——这与 edgecore 内部 metamanager 持久化资源的方式完全一致因此读到的就是边缘侧“当前视图”。Pod 状态的拼接getPodsFromDatabaseget.go#L297-L341会先查询pod类型记录再按 key 规则把pod替换为podstatus查出对应的PodStatusRequest将其中Status字段合并进 Pod 的status后再输出。这解释了为什么表格中STATUS/READY列有值——它们来自独立的 podstatus 记录而非 Pod 对象本身。输出渲染表格输出复用了 kubectl 的printers/TableConvertor体系get.go#L642-L648 的ConvertDataToTable列定义与 api-server 的 Table 输出保持一致-o json|yaml则走JSONYamlPrint将 Meta 还原为标准v1.Pod/v1.Service等对象后序列化。get_flags.go 的文件头注释也说明了这一点该文件衍生自 k8s kubectl 的get_flags.go并做了精简。标签过滤-l选择器由SplitSelectorParameters解析支持、与!三种比较get.go#L502-L542过滤发生在数据库查询结果之上基于对象metadata.labels。keadm debug check检查节点是否满足 edgecore 运行条件功能与用法check用于验证系统特定项是否满足 edgecore 安装与运行的要求覆盖 CPU、内存、磁盘、DNS、网络、PID 限制和容器运行时七类检查项可用all一次执行全部Usage: keadm debug check [command] Examples: # Check all items . keadm debug check all # Check whether the node CPU meets requirements. keadm debug check cpu # Check whether the node memory meets requirements. keadm debug check mem # check whether the node disk meets requirements. keadm debug check disk # Check whether the node DNS can resolve a specific domain name. keadm debug check dns -d www.github.com # Check whether the node network meets requirements. keadm debug check network # Check whether the number of free processes on the node meets requirements. keadm debug check pid # Check whether runtime(Docker) is installed on the node. keadm debug check runtime Available Commands: all Check all item cpu Check node CPU requirements disk Check node disk requirements dns Check whether DNS can work mem Check node memory requirements network Check whether the network is normal pid Check node PID requirements runtime Check whether runtime can work各检查项的判定阈值从源码 check.go 与阈值常量 constant.go#L219-L222 可以确认每项的判定标准子命令判定条件关键参数cpu核心数 ≥ 1 vCore且 CPU 使用率 90%无mem内存总量 256MB空闲内存 128MB使用率 90%无disk磁盘总量 1GB空闲 512MB使用率 90%无dns域名可解析默认测试域名为www.github.com-d指定域名-D指定 DNS 服务器 IPnetworkping 测试 IP 零丢包HTTPS 连通 cloudhubHTTP 连通本地127.0.0.1:10350-i指定测试 IP缺省自动取/etc/resolv.conf中的 nameserver-s指定 cloudhub 地址缺省从 edgecore 配置读取-c指定配置文件pid空闲进程比例 (1 - 当前进程数/最大 PID 数) 5%无runtime容器运行时可连接且处于运行状态-c指定 edgecore 配置文件从源码实现看几个细节check dns指定 DNS 服务器CheckDNSSpecify 通过替换net.DefaultResolver的 Dial 函数强制向指定IP:53的 UDP 端口发起解析从而验证“某个 DNS 服务器能否解析某个域名”——这在验证集群内 kube-dnsCoreDNS时非常有用。check network的三级验证CheckNetWork 依次执行 ①ping测试 IP未指定时通过 CmdGetDNSIP 从/etc/resolv.conf提取 nameserver② 对 edgecore 配置中edgehub.webSocket.server发起 HTTPS 请求CheckHTTP 对 x509 证书错误视为可达因为网络层已通③ 对127.0.0.1:10350EdgeCoreServer 常量发起 HTTP 请求验证本机 edgecore 服务存活。check runtime的运行时来源提案中曾设想用-r参数指定容器运行时从当前源码看getRuntimeEndpointcheck.go#L375-L402改为优先从 edgecore 配置modules.edged.tailoredKubeletConfig.containerRuntimeEndpoint读取运行时端点未配置时回退到默认 remote runtime 端点随后复用 kubeadm 的utilruntime.NewContainerRuntime完成 Connect IsRunning 两步检查。check pidCheckPid 通过 shell 读取系统最大进程数与当前进程数空闲比例不足 5% 即判定失败这与提案中“可用进程数少于 5% 视为不足”的描述一致。硬件指标采集统一使用gopsutilcpu/mem/disk三个包不依赖特定发行版的命令输出格式。实际运行示例提案原文示例[rootlocalhost tmp]# keadm debug check cpu CPU total: 1 core, Allowed 1 core CPU usage rate: 0.00, Allowed rate 0.9 |-----------------| ||check cpu succeed| |-----------------|[rootlocalhost tmp]# keadm debug check mem Memory total: 1833.33 MB, Allowed 256 MB Memory Free total: 1074.40 MB, Allowed 128 MB Memory usage rate: 0.11, Allowed rate 0.9 |-----------------| ||check mem succeed| |-----------------|[rootlocalhost tmp]# keadm debug check disk Disk total: 50268.47 MB, Allowed 1024 MB Disk Free total: 39087.80 MB, Allowed 512MB Disk usage rate: 0.18, Allowed rate 0.9 |------------------| ||check disk succeed| |------------------|[rootlocalhost tmp]# keadm debug check dns dns resolution success, domain: www.github.com ip: 192.30.255.113 |-----------------| ||check dns succeed| |-----------------|验证集群内 K8s Service DNS 解析先取得 kube-dns 的 ClusterIP 作为 DNS 服务器[rootk8s ~]# kubectl get svc -n kube-system |grep dns kube-dns ClusterIP 10.96.0.10 none 53/UDP,53/TCP,9153/TCP 118d [rootk8s ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 none 443/TCP 118d nginx ClusterIP 10.97.180.216 none 80/TCP 73d [rootlocalhost tmp]# keadm debug check dns -D 10.96.0.10 -d nginx dns resolution success, domain: nginx ip: 10.97.180.216 |-----------------| ||check dns succeed| |-----------------|[rootlocalhost tmp]# keadm debug check network ping 172.17.0.1 success check cloudhubServer 10.211.55.6:10000 success check edgecoreServer 127.0.0.1:10350 success |---------------------| ||check network succeed| |---------------------|Maximum PIDs: 32768; Running processes: 129 |-----------------| ||check pid succeed| |-----------------|[rootlocalhost tmp]# keadm debug check runtime docker is running |---------------------| ||check runtime succeed| |---------------------|check all汇总输出[rootlocalhost tmp]# keadm debug check all CPU total: 1 core, Allowed 1 core CPU usage rate: 0.10, Allowed rate 0.9 Memory total: 1833.33 MB, Allowed 256 MB Memory Free total: 605.50 MB, Allowed 128 MB Memory usage rate: 0.17, Allowed rate 0.9 Disk total: 50268.47 MB, Allowed 1024 MB Disk Free total: 39085.29 MB, Allowed 512MB Disk usage rate: 0.18, Allowed rate 0.9 dns resolution success, domain: www.github.com ip: 192.30.255.112 ping 172.17.0.1 success check cloudhubServer 10.211.55.6:10000 success check edgecoreServer 127.0.0.1:10350 success Maximum PIDs: 32768; Running processes: 129 docker is running |-----------------| ||check all succeed| |-----------------|keadm debug collect一键打包节点全量诊断数据功能与用法collect将当前节点上与 KubeEdge 排查相关的全部数据收集、压缩为edge-时间戳.tar.gz供运维人员离线分析Usage: keadm debug collect [flags] Examples: keadm debug collect --path . # Collect all items and specified the output directory path keadm debug collect --output-path . Flags: -c, --config string Specify configuration file, default is /etc/kubeedge/config/edgecore.yaml -d, --detail Whether to print internal log output -h, --help help for collect -l, --log-path string Specify log file (default /var/log/kubeedge/) -o, --output-path string Cache data and store data compression packages in a directory that default to the current directory参数说明与 collect.go 中的 flag 定义一致-c, --configedgecore 配置文件默认/etc/kubeedge/config/edgecore.yaml。源码中VerificationParameters会校验该文件存在不存在直接报错-d, --detail打印每一步的内部执行日志创建临时目录、每条 shell 命令、每个文件复制-l, --log-path日志目录默认/var/log/kubeedge/-o, --output-path压缩包的输出目录默认当前目录.。注意源码会将其解析为绝对路径并要求该目录已存在否则会报output-path ... does not exist。收集范围按提案与源码 collect.go数据分为三大类1. 系统数据collectSystemDatacollect.go#L157-L206数据项采集方式硬件架构archCPU 信息复制/proc/cpuinfo内存信息复制/proc/meminfo磁盘信息df -h网络信息netstat -anp复制/etc/resolv.conf与/etc/hosts进程信息ps -aux时间信息date与uptime历史命令history -a cat ~/.bash_history2. Edgecore 数据collectEdgecoreDatacollect.go#L209-L268数据库文件优先取配置中dataBase.dataSource缺省为/var/lib/kubeedge/edgecore.db日志-l指定的目录或默认/var/log/kubeedge/systemd 服务文件/lib/systemd/system/edgecore.service配置文件目录/etc/kubeedge/config/证书优先取配置中modules.edgeHub.tlsCertFile/tlsPrivateKeyFile与tlsCaFile缺省回退到/etc/kubeedge/certs/与默认 CA 路径软件版本执行edgecore --version。3. 容器运行时数据源码中定义了collectRuntimeDatacollect.go#L271-L290用于收集docker version、docker info、docker images、docker ps -a、runtime 日志及docker.service等。从当前源码结构看ExecuteCollect流程中该步骤尚带// TODO: collectRuntimeData with containerd的适配注释因此不同版本中 runtime 部分可能未出现在压缩包里使用时以实际输出为准。实际运行示例提案原文示例默认收集[rootlocalhost tmp]# keadm debug collect Start collecting data Data collected successfully, path: /root/tmp/edge_2020_1109_173744.tar.gz-d打开明细可看到完整的采集过程临时目录位于/tmp/edge_时间戳按 system/edgecore/runtime 分子目录组织[rootlocalhost tmp]# keadm debug collect -d Start collecting data create tmp file: /tmp/edge_2020_1109_173800 create tmp file: /tmp/edge_2020_1109_173800/system Execute Shell: arch /tmp/edge_2020_1109_173800/system/arch Copy File: cp -r /proc/cpuinfo /tmp/edge_2020_1109_173800/system/ Copy File: cp -r /proc/meminfo /tmp/edge_2020_1109_173800/system/ Execute Shell: df -h /tmp/edge_2020_1109_173800/system/disk Copy File: cp -r /etc/hosts /tmp/edge_2020_1109_173800/system/ Copy File: cp -r /etc/resolv.conf /tmp/edge_2020_1109_173800/system/ Execute Shell: ps -axu /tmp/edge_2020_1109_173800/system/process Execute Shell: date /tmp/edge_2020_1109_173800/system/date Execute Shell: uptime /tmp/edge_2020_1109_173800/system/uptime Execute Shell: history -a cat ~/.bash_history /tmp/edge_2020_1109_173800/system/history Execute Shell: netstat -pan /tmp/edge_2020_1109_173800/system/network collect systemd data finish create tmp file: /tmp/edge_2020_1109_173800/edgecore Copy File: cp -r /var/lib/kubeedge/edgecore.db /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /var/log/kubeedge/ /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /lib/systemd/system/edgecore.service /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/config/ /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/certs/server.crt /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/certs/server.key /tmp/edge_2020_1109_173800/edgecore/ Copy File: cp -r /etc/kubeedge/ca/rootCA.crt /tmp/edge_2020_1109_173800/edgecore/ Execute Shell: edgecore --version /tmp/edge_2020_1109_173800/edgecore/version collect edgecore data finish create tmp file: /tmp/edge_2020_1109_173800/runtime Copy File: cp -r /lib/systemd/system/docker.service /tmp/edge_2020_1109_173800/runtime/ Execute Shell: docker version /tmp/edge_2020_1109_173800/runtime/version Execute Shell: docker info /tmp/edge_2020_1109_173800/runtime/info Execute Shell: docker images /tmp/edge_2020_1109_173800/runtime/images Execute Shell: docker ps -a /tmp/edge_2020_1109_173800/runtime/containerInfo Execute Shell: journalctl -u docker /tmp/edge_2020_1109_173800/runtime/log collect runtime data finish Data compressed successfully Remove tmp data finish Data collected successfully, path: /root/tmp/edge_2020_1109_173800.tar.gz指定输出目录与配置文件[rootlocalhost tmp]# keadm debug collect -o /tmp Start collecting data Data collected successfully, path: /tmp/edge_2020_1109_173842.tar.gz[rootlocalhost tmp]# keadm debug collect -c /etc/kubeedge/config/edgecore.yaml Start collecting data Data collected successfully, path: /root/tmp/edge_2020_1109_173931.tar.gz整体流程ExecuteCollect为参数校验 → 在/tmp下创建edge_时间戳临时目录 → 采集系统数据 → 解析 edgecore 配置并采集 edgecore 数据 →util.Compress打包为edge_时间戳.tar.gz→ 清理临时目录。注意压缩包中会包含证书文件提交给他人排查时请知悉其中的敏感信息。keadm debug diagnose面向故障场景的诊断功能与用法diagnose针对特定故障场景做“结论式”诊断直接输出成功/失败Usage: keadm debug diagnose [command] Examples: # Diagnose whether the node is normal keadm debug diagnose node # Diagnose whether the pod is normal keadm debug diagnose pod nginx-xxx -n test # Diagnose node installation conditions keadm debug diagnose install # Diagnose node installation conditions and specify the detected ip keadm debug diagnose install -i 192.168.1.2 Available Commands: install Diagnose install node Diagnose edge node pod Diagnose pod各子命令的诊断范围与源码行为diagnose nodeDiagnoseNode依次检查edgecore 进程是否正在运行edgecore 配置文件是否存在默认/etc/kubeedge/config/edgecore.yaml可用-c覆盖数据库文件是否存在配置dataBase.dataSource缺省/var/lib/kubeedge/edgecore.dbedgehub websocket 是否启用并对配置中的 cloudhub 地址发起 HTTPS 连通性检查。diagnose pod name [-n ns]源码中diagnose pod会先执行完整的 node 诊断diagnose.go#L96-L105通过后打开本地数据库按 key 规则查询ns/pod/name与ns/podstatus/name两条记录检查 Pod phase、ConditionsReady 等与容器状态Waiting/Terminated 的 message、reason、RestartCount最终给出Pod xxx is Ready或失败原因。diagnose install与keadm debug check all语义相近DiagnoseInstall依次执行 CPU、内存、磁盘、DNS、网络、PID 检查用于安装前/重装前验证节点条件。参数-i指定测试 IP、-d指定测试域名、-D指定 DNS 服务器 IP、-s指定 cloudhub 地址。实际运行示例提案原文示例节点健康[rootlocalhost tmp]# keadm debug diagnose node edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection success |---------------------| ||diagnose node succeed| |---------------------|Pod 诊断成功[rootlocalhost tmp]# keadm debug diagnose pod nginx-ds-85jch -n default edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection successDatabase /var/lib/kubeedge/edgecore.db is exist Pod nginx-ds-85jch is exist PodStatus nginx-ds-85jch is exist pod nginx-ds-85jch phase is Running containerConditions nginx-ds is ready Pod nginx-ds-85jch is Ready |--------------------| ||diagnose pod succeed| |--------------------|Pod 在指定 namespace 下不存在[rootlocalhost tmp]# keadm debug diagnose pod nginx-ds-85jch -n kube-system edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection successDatabase /var/lib/kubeedge/edgecore.db is exist not find kube-system/pod/nginx-ds-85jch in database |-------------------| ||diagnose pod failed| |-------------------|Pod 存在但容器无法启动典型的故障定位输出直接给出 OCI 报错原因与重启次数[rootlocalhost tmp]# keadm debug diagnose pod nginx-deployment-dbbffc676-wprs8 edgecore is running edge config is exists: /etc/kubeedge/config/edgecore.yaml docker is running dataSource is exists: /var/lib/kubeedge/edgecore.db cloudcore websocket connection successDatabase /var/lib/kubeedge/edgecore.db is exist Pod nginx-deployment-dbbffc676-wprs8 is exist PodStatus nginx-deployment-dbbffc676-wprs8 is exist pod nginx-deployment-dbbffc676-wprs8 phase is Running conditions is not true, type: Ready ,message: containers with unready status: [nginx] ,reason: ContainersNotReady containerConditions nginx Terminated, message: oci runtime error: container_linux.go:235: starting container process caused exec: \/abc\: stat /abc: no such file or directory , reason: ContainerCannotRun, RestartCount: 104 Pod nginx-deployment-dbbffc676-wprs8 is not Ready |-------------------| ||diagnose pod failed| |-------------------|可以看到诊断输出直接定位到“容器 entrypoint/abc不存在”这一根因这正是提案希望达成的“定位故障原因”效果。推荐的排障工作流结合四个命令的定位建议按以下顺序在边缘节点上排障快速健康检查keadm debug diagnose node——确认 edgecore 进程、配置、数据库、到 cloudhub 的连通性四项基本盘环境与安装条件keadm debug check all——确认资源、DNS、网络、PID、运行时是否满足 edgecore 运行要求部署新节点时用keadm debug diagnose install做前置验证资源层面核对keadm debug get pod -A/keadm debug get all -o yaml——确认云侧下发的资源是否已同步到本地数据库Pod 状态停在什么阶段单点深入keadm debug diagnose pod name -n ns——拿到 Ready 条件与容器级报错留档求助keadm debug collect -d——打包节点全量数据交给进一步分析。适用前提与注意事项命令需在边缘节点本机执行且建议以 root或加sudo运行因为 collect 需要读取系统文件与证书目录get/diagnose的数据源是 edgecore 本地数据库默认路径/var/lib/kubeedge/edgecore.db可用-p覆盖若 edgecore 配置中dataBase.dataSource指向其他路径diagnose 会以配置为准check network与diagnose node的 cloudhub 地址默认从 edgecore 配置默认/etc/kubeedge/config/edgecore.yaml中的modules.edgeHub.webSocket.server读取也可用-s显式覆盖本文示例输出取自提案文档 keadm-debug.md 中的实测记录不同 KubeEdge 版本的实际命令输出格式可能略有差异参数与判定逻辑以当前仓库 keadm/cmd/keadm/app/cmd/debug/ 下的源码为准单元测试可参考 check_test.go、collect_test.go、diagnose_test.go 与 get_flags_test.go。【免费下载链接】kubeedgeKubernetes Native Edge Computing Framework (project under CNCF)项目地址: https://gitcode.com/GitHub_Trending/ku/kubeedge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考