基于 agents24 可观测性插件的生产级监控体系搭建指南Prometheus、Grafana、链路追踪、日志与 SLO 全栈实践【免费下载链接】agentsMulti-harness agentic plugin marketplace for Claude Code, Codex, Cursor, OpenCode, GitHub Copilot, and Google Antigravity项目地址: https://gitcode.com/GitHub_Trending/agents24/agents本篇技术指南围绕 agents24 仓库中observability-monitoring插件的monitor-setup命令展开完整讲解如何在 Kubernetes 与微服务场景下搭建指标Metrics 日志Logs 链路追踪Traces三支柱可观测体系并落地 Grafana 仪表盘、多路告警、SLO 错误预算与 Terraform 基础设施即代码。读完本文你将掌握一套可直接复制的监控基础设施配置模板、应用埋点代码范式以及从采集到告警再到 SLO 治理的完整链路设计方法。命令定位monitor-setup 能交付什么monitor-setup是observability-monitoring插件提供的核心命令之一。在已安装该插件的 Claude Code 等 harness 环境中可以通过斜杠命令直接调用调用格式详见 docs/usage.md/observability-monitoring:monitor-setup 为 50 微服务设计完整监控方案命令以user_request标签内的文本作为交付目标描述并要求 Agent 扮演监控与可观测性专家围绕三支柱进行现状评估、架构设计、落地实施与告警策略制定。同插件还提供slo-implement命令见 plugins/observability-monitoring/commands/slo-implement.md用于 SLO 专项落地以及四个配套 Agentobservability-engineer、performance-engineer、network-engineer、database-optimizer与四个技能prometheus-configuration、grafana-dashboards、distributed-tracing、slo-implementation共同构成完整的运维可观测能力面。按 docs/architecture.md 的多插件组合范式monitor-setup通常作为完整交付流程的最后一环例如backend-development:feature-development→security-scanning:security-hardening→comprehensive-review:full-review→cicd-automation:workflow-automate→observability-monitoring:monitor-setup用于在功能上线前补齐监控能力。命令最终要求输出 8 类交付物详见文末交付物清单基础设施现状评估、监控架构设计、分步部署方案、指标目录、可复用仪表盘、告警运行手册、SLO 定义与接入指南。一、Prometheus 指标采集从抓取配置到自定义埋点Prometheus 采用拉取pull模型应用通过/metrics端点暴露指标Prometheus 周期性抓取并存储于本地 TSDB。monitor-setup 给出了完整的prometheus.yml与 Node.js 侧的自定义指标实现。1.1 全局与抓取配置# prometheus.yml global: scrape_interval: 15s evaluation_interval: 15s external_labels: cluster: production region: us-east-1 alerting: alertmanagers: - static_configs: - targets: [alertmanager:9093] rule_files: - alerts/*.yml - recording_rules/*.yml scrape_configs: - job_name: prometheus static_configs: - targets: [localhost:9090] - job_name: node static_configs: - targets: [node-exporter:9100] - job_name: application kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true逐项解读其中的关键参数global.scrape_interval15s对所有未单独指定抓取间隔的 job 生效的默认采集周期。global.evaluation_interval15s则是告警规则与录制规则recording rule的默认评估周期。skill 文档建议典型区间为 15–60s见 plugins/observability-monitoring/skills/prometheus-configuration/SKILL.md。external_labels在抓取到的所有样本上附加的全局标签用于区分多集群/多区域部署是后续在 Grafana 中按cluster、region筛选与做跨集群联邦的关键。alerting.alertmanagers把告警推送到alertmanager:9093配合后面的 Alertmanager 配置实现路由与通知。rule_files声明式加载告警规则alerts/*.yml与录制规则recording_rules/*.yml使规则文件与主配置分离、便于版本管理。scrape_configs三种典型 job——prometheus抓取自身/metricslocalhost:9090用于自监控如up{jobprometheus}node抓取 node-exporter:9100获得主机级 CPU、内存、磁盘、网络指标application通过kubernetes_sd_configs的role: pod做 Kubernetes Pod 服务发现并用 relabel 规则keep只保留带有prometheus.io/scrape: true注解的 Pod——这是社区标准的注解即接入模式。关于 Kubernetes 服务发现与 relabel 的完整示例包括__meta_kubernetes_pod_annotation_prometheus_io_path覆盖/metrics路径、端口替换、注入namespace/pod标签可进一步参考 plugins/observability-monitoring/skills/prometheus-configuration/references/details.md该技能还给出静态目标、基于文件的file_sd_configs服务发现等替代方案。1.2 自定义业务指标prom-client 埋点对应用侧指标monitor-setup 给出基于prom-client的 TypeScript 采集器实现// metrics.ts import { Counter, Histogram, Gauge, Registry } from prom-client; export class MetricsCollector { private registry: Registry; private httpRequestDuration: Histogramstring; private httpRequestTotal: Counterstring; constructor() { this.registry new Registry(); this.initializeMetrics(); } private initializeMetrics() { this.httpRequestDuration new Histogram({ name: http_request_duration_seconds, help: Duration of HTTP requests in seconds, labelNames: [method, route, status_code], buckets: [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2, 5], }); this.httpRequestTotal new Counter({ name: http_requests_total, help: Total number of HTTP requests, labelNames: [method, route, status_code], }); this.registry.registerMetric(this.httpRequestDuration); this.registry.registerMetric(this.httpRequestTotal); } httpMetricsMiddleware() { return (req: Request, res: Response, next: NextFunction) { const start Date.now(); const route req.route?.path || req.path; res.on(finish, () { const duration (Date.now() - start) / 1000; const labels { method: req.method, route, status_code: res.statusCode.toString(), }; this.httpRequestDuration.observe(labels, duration); this.httpRequestTotal.inc(labels); }); next(); }; } async getMetrics(): Promisestring { return this.registry.metrics(); } }要点说明指标命名规范http_request_duration_seconds遵循前缀_对象_单位命名时间类指标统一使用秒并加_seconds后缀这是 prometheus-configuration 技能 Best Practices 第一条见 SKILL.md保证指标可被下游工具正确解析与换算。三类核心指标类型Counter只增不减适合请求总数、Histogram带分桶用于计算 p50/p95/p99 时延代码中的buckets: [0.001, 0.005, ..., 5]定义了从 1ms 到 5s 的分桶边界、Gauge可增可减适合当前值类指标。标签labels设计method、route、status_code三个标签维度支撑了后续仪表盘按方法拆分请求速率、按状态码计算错误率的能力但要注意高基数标签如 userId会显著增加存储成本。res.on(finish)钩子在响应结束时统一记录耗时与总数覆盖正常与异常响应getMetrics()暴露的registry.metrics()文本格式正是/metrics端点可直接返回的 Prometheus 暴露格式。1.3 安装与上线验证结合 prometheus-configuration 技能Kubernetes 场景推荐用 Helm 一键部署kube-prometheus-stack内置 Prometheus、Alertmanager、Grafana 与常用告警规则helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install prometheus prometheus-community/kube-prometheus-stack \ --namespace monitoring \ --create-namespace \ --set prometheus.prometheusSpec.retention30d \ --set prometheus.prometheusSpec.storageVolumeSize50Gi配置完成后用以下命令验证抓取目标、配置生效与查询可用性见 SKILL.md 与 details.md# 查看抓取目标与健康状态 curl http://localhost:9090/api/v1/targets # 查看当前生效配置 curl http://localhost:9090/api/v1/status/config # 验证一条查询 curl http://localhost:9090/api/v1/query?queryup # promtool 离线校验配置与规则 promtool check config prometheus.yml promtool check rules /etc/prometheus/rules/*.yml二、Grafana 仪表盘以 Golden Signals 为核心的速效面板monitor-setup 给出的仪表盘生成器以黄金信号Golden Signals为骨架——请求速率Rate、错误率Errors、时延分位数Duration三块面板排布在第一行// dashboards/service-dashboard.ts export const createServiceDashboard (serviceName: string) { return { title: ${serviceName} Service Dashboard, uid: ${serviceName}-overview, tags: [service, serviceName], time: { from: now-6h, to: now }, refresh: 30s, panels: [ // Golden Signals { title: Request Rate, type: graph, gridPos: { x: 0, y: 0, w: 6, h: 8 }, targets: [ { expr: sum(rate(http_requests_total{service${serviceName}}[5m])) by (method), legendFormat: {{method}}, }, ], }, { title: Error Rate, type: graph, gridPos: { x: 6, y: 0, w: 6, h: 8 }, targets: [ { expr: sum(rate(http_requests_total{service${serviceName},status_code~5..}[5m])) / sum(rate(http_requests_total{service${serviceName}}[5m])), legendFormat: Error %, }, ], }, { title: Latency Percentiles, type: graph, gridPos: { x: 12, y: 0, w: 12, h: 8 }, targets: [ { expr: histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service${serviceName}}[5m])) by (le)), legendFormat: p50, }, { expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service${serviceName}}[5m])) by (le)), legendFormat: p95, }, { expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service${serviceName}}[5m])) by (le)), legendFormat: p99, }, ], }, ], }; };关键 PromQL 解读请求速率sum(rate(http_requests_total[5m])) by (method)计算 5 分钟窗口的每秒请求数按 HTTP 方法聚合。rate()只对 Counter 类型有意义且窗口通常取抓取间隔的 3 倍以上15s 抓取 → 推荐 ≥1m以保证平滑。错误率分子用正则status_code~5..匹配 5xx 服务端错误除以总请求得到错误占比——这是 SLO 中 Availability SLI 的直接输入。时延分位数histogram_quantile(0.95, sum(rate(..._bucket[5m])) by (le))基于直方图分桶_bucket系列指标估算 p95。注意必须先sum by (le)把所有维度归并到le上限桶后再对同一组le序列求分位数。面板参数中gridPos控制 24 列栅格布局三个面板 6/6/12 宽、time与refresh设定默认时间窗口近 6 小时与自动刷新频率30s。monitor-setup 只是给了黄金信号起点。若需要更完整的设计能力grafana-dashboards 技能SKILL.md补充了以下可直接复用的扩展设计原则信息层级大数字关键指标 → 时序趋势 → 明细表/热力图服务侧用RED 方法Rate/Errors/Duration资源侧用USE 方法Utilization/Saturation/Errors。面板类型stat单值卡、graph时序图、table表配合up指标 重命名变换、heatmap时延热力图数据格式tsbuckets。模板变量通过templating.list定义namespace/service下拉变量label_values(kube_pod_info, namespace)并在面板表达式中以$namespace、$service引用一份仪表盘即可复用到任意服务。仪表盘即代码提供 Grafana provisioning 的dashboards.ymltype: file、path: /etc/grafana/dashboards以及 Terraform 的grafana_dashboard资源写法与本文第七节的 IaC 思路一致。面板内联告警alert块可对错误率面板配置5%且持续 5 分钟触发、通知到 Slack channel。三、分布式链路追踪OpenTelemetry Jaeger 的请求全链路可视monitor-setup 给出基于 OpenTelemetry Node SDK 的追踪初始化实现// tracing.ts import { NodeSDK } from opentelemetry/sdk-node; import { getNodeAutoInstrumentations } from opentelemetry/auto-instrumentations-node; import { Resource } from opentelemetry/resources; import { SemanticResourceAttributes } from opentelemetry/semantic-conventions; import { JaegerExporter } from opentelemetry/exporter-jaeger; import { BatchSpanProcessor } from opentelemetry/sdk-trace-base; export class TracingSetup { private sdk: NodeSDK; constructor(serviceName: string, environment: string) { const jaegerExporter new JaegerExporter({ endpoint: process.env.JAEGER_ENDPOINT || http://localhost:14268/api/traces, }); this.sdk new NodeSDK({ resource: new Resource({ [SemanticResourceAttributes.SERVICE_NAME]: serviceName, [SemanticResourceAttributes.SERVICE_VERSION]: process.env.SERVICE_VERSION || 1.0.0, [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: environment, }), traceExporter: jaegerExporter, spanProcessor: new BatchSpanProcessor(jaegerExporter), instrumentations: [ getNodeAutoInstrumentations({ opentelemetry/instrumentation-fs: { enabled: false }, }), ], }); } start() { this.sdk .start() .then(() console.log(Tracing initialized)) .catch((error) console.error(Error initializing tracing, error)); } shutdown() { return this.sdk.shutdown(); } }实现要点Resource 语义化标签通过SemanticResourceAttributes.SERVICE_NAME / SERVICE_VERSION / DEPLOYMENT_ENVIRONMENT注入服务名、版本、环境。服务名是 Jaeger 检索与依赖图service dependency graph的聚合键版本标签则支持发布后按版本对比延迟回归。自动埋点getNodeAutoInstrumentations()一键接入 HTTP、Express、gRPC、数据库客户端等常用库的自动埋点并显式关闭instrumentation-fs文件系统操作噪音大、价值低。distributed-tracing 技能SKILL.md同时给出生产建议采样率 1–10%、合理使用标签、跨服务传播上下文、关注追踪开销目标 1% CPU。BatchSpanProcessor批量异步导出 Span避免每个请求都同步阻塞网络 I/O是高流量下的标准选择distributed-tracing 技能的排查清单也把使用批量 Span 处理器列为高延迟开销的修复手段之一。3.1 追踪链路的结构化理解distributed-tracing 技能定义了完整的 Trace 模型见 references/details.mdTrace (Request ID: abc123) ↓ Span (frontend) [100ms] ↓ Span (api-gateway) [80ms] ├→ Span (auth-service) [10ms] └→ Span (user-service) [60ms] └→ Span (database) [40ms]Trace一次端到端请求的完整旅程Span旅程中的单个操作Context跨服务传播的元数据标准 W3C 头为traceparent、tracestateTags用于过滤的键值对LogsSpan 内带时间戳的事件。上下文传播是跨服务关联的关键。Python 侧使用opentelemetry.propagate.inject(headers)把当前 trace 上下文注入出站请求头Node.js 侧使用propagation.inject(context.active(), headers)保证下游服务能续接同一 Trace。3.2 Jaeger 部署与采样策略技能同时给出 Jaeger 的本地/集群部署方式Docker Compose 暴露6831UDP agent 端口、14268HTTP collector、16686UI 与9411Zipkin 兼容端口以及三类采样策略# 概率采样采样 1% 的 trace sampler: type: probabilistic param: 0.01 # 限流采样每秒最多采样 100 条 sampler: type: ratelimiting param: 100生产环境常用ParentBased(rootTraceIdRatioBased(0.01))的父级采样一条 trace 是否被采样由根 Span 决定从而保证整条链路要么完整保留、要么完整丢弃避免出现半截 trace。上线后排查两类典型问题见 SKILL.md没有 trace——检查 collector 端点、网络连通性、采样配置与应用日志追踪开销过高——降低采样率、改用批量处理器、检查 exporter 配置。四、日志聚合Fluentd 采集管线与结构化日志4.1 Fluentd 采集与加工monitor-setup 给出基于 Fluentd 的 Kubernetes 容器日志采集配置# fluent.conf source type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* parse type json time_format %Y-%m-%dT%H:%M:%S.%NZ /parse /source filter kubernetes.** type kubernetes_metadata kubernetes_url #{ENV[KUBERNETES_SERVICE_HOST]} /filter filter kubernetes.** type record_transformer record cluster_name ${ENV[CLUSTER_NAME]} environment ${ENV[ENVIRONMENT]} timestamp ${time.strftime(%Y-%m-%dT%H:%M:%S.%LZ)} /record /filter match kubernetes.** type elasticsearch host #{ENV[FLUENT_ELASTICSEARCH_HOST]} port #{ENV[FLUENT_ELASTICSEARCH_PORT]} index_name logstash logstash_format true buffer type file path /var/log/fluentd-buffers/kubernetes.buffer flush_interval 5s chunk_limit_size 2M /buffer /match配置语义拆解sourcetail以 tail 方式监听/var/log/containers/*.logKubernetes 容器标准日志路径用pos_file记录读取位置实现断点续读tag kubernetes.*为后续 filter/match 的路由匹配做准备parse指定 JSON 解析与时间格式纳秒精度%NZ。filter kubernetes_metadata为每条日志附加 Pod 元数据命名空间、Pod 名、容器名、标签等是实现按服务/命名空间检索日志的基础。filter record_transformer注入cluster_name、environment等环境标签并统一重写timestamp为毫秒级 ISO8601——这是日志时间字段标准化的关键一步保证 Elasticsearch/Kibana 的时间索引正确。match elasticsearch输出到 ESlogstash_format true按天自动生成logstash-YYYY.MM.DD索引buffer定义落盘缓冲每 5s flush、单块上限 2M保证 ES 短暂不可用时日志不丢、且避免频繁小批量写入。4.2 应用侧结构化日志Python仅靠采集端加工还不够应用侧必须以结构化 JSON 输出日志才能被type json解析并保留字段语义# structured_logging.py import json import logging from datetime import datetime from typing import Any, Dict, Optional class StructuredLogger: def __init__(self, name: str, service: str, version: str): self.logger logging.getLogger(name) self.service service self.version version self.default_context { service: service, version: version, environment: os.getenv(ENVIRONMENT, development) } def _format_log(self, level: str, message: str, context: Dict[str, Any]) - str: log_entry { timestamp: datetime.utcnow().isoformat() Z, level: level, message: message, **self.default_context, **context } trace_context self._get_trace_context() if trace_context: log_entry[trace] trace_context return json.dumps(log_entry) def info(self, message: str, **context): log_msg self._format_log(INFO, message, context) self.logger.info(log_msg) def error(self, message: str, error: Optional[Exception] None, **context): if error: context[error] { type: type(error).__name__, message: str(error), stacktrace: traceback.format_exc() } log_msg self._format_log(ERROR, message, context) self.logger.error(log_msg)设计亮点统一字段基线每条日志都带timestamp、level、message并默认注入service、version、environment上下文与 Fluentd 的record_transformer补充字段呼应。Trace 关联_get_trace_context()尝试把当前 OpenTelemetry 的 trace_id 写入trace字段。这正是日志与链路打通的关键——Kibana 中按 trace_id 检索即可把一条业务日志对应到整条请求链路。distributed-tracing 技能给出了具体实现范式SKILL.mdtrace.get_current_span()拿到 span 上下文后将format(trace_id, 032x)写入日志 extra。异常结构化error()自动把异常类型、消息与堆栈序列化进error对象避免异常信息散落在多行文本中。五、告警配置从 Prometheus 规则到 Alertmanager 路由5.1 告警规则monitor-setup 给出应用层与基础设施层两组规则# alerts/application.yml groups: - name: application interval: 30s rules: - alert: HighErrorRate expr: | sum(rate(http_requests_total{status_code~5..}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) 0.05 for: 5m labels: severity: critical annotations: summary: High error rate on {{ $labels.service }} description: Error rate is {{ $value | humanizePercentage }} - alert: SlowResponseTime expr: | histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le) ) 1 for: 10m labels: severity: warning annotations: summary: Slow response time on {{ $labels.service }} - name: infrastructure rules: - alert: HighCPUUsage expr: avg(rate(container_cpu_usage_seconds_total[5m])) by (pod) 0.8 for: 15m labels: severity: warning - alert: HighMemoryUsage expr: | container_memory_working_set_bytes / container_spec_memory_limit_bytes 0.9 for: 10m labels: severity: critical规则编写要点for子句是告警去抖的核心表达式持续满足该时长才真正触发例如错误率 5% 需持续 5 分钟、P95 时延 1s 需持续 10 分钟避免瞬时抖动刷屏。这与 grafana-dashboards 技能中面板内联告警的for: 5m同理。labels.severity用于区分 critical/warning是 Alertmanager 路由分发的输入。annotations支持模板变量{{ $labels.service }}引用触发实例的标签、{{ $value | humanizePercentage }}把 PromQL 数值格式化为百分比保证告警通知携带可读上下文。表达式语义HighErrorRate是5xx 速率 / 总请求速率 0.05SlowResponseTime复用第一节的直方图分位数表达式容器 CPU/内存告警分别以 80%15 分钟与 90%10 分钟为阈值。prometheus-configuration 技能的 告警规则参考 还补充了服务宕机up{jobmy-app} 0、基于录制规则的错误率百分比、磁盘水位等常见模板可直接并入alerts/目录。5.2 Alertmanager 路由与通知# alertmanager.yml global: resolve_timeout: 5m slack_api_url: $SLACK_API_URL route: group_by: [alertname, cluster, service] group_wait: 10s group_interval: 10s repeat_interval: 12h receiver: default routes: - match: severity: critical receiver: pagerduty continue: true - match_re: severity: critical|warning receiver: slack receivers: - name: slack slack_configs: - channel: #alerts title: {{ .GroupLabels.alertname }} text: {{ range .Alerts }}{{ .Annotations.description }}{{ end }} send_resolved: true - name: pagerduty pagerduty_configs: - service_key: $PAGERDUTY_SERVICE_KEY description: {{ .GroupLabels.alertname }}: {{ .Annotations.summary }}路由策略解读分组group_by: [alertname, cluster, service]把同一服务同类告警合并为一条通知group_wait: 10s等待同一组内告警聚齐再发送repeat_interval: 12h控制未解决告警的重复提醒频率防止疲劳轰炸。子路由匹配severity: critical的告警同时发给 PagerDutycontinue: true表示继续匹配后续规则与 Slackwarning 及以上通过正则match_re进 Slack。由此形成紧急走电话、一般走 IM的经典分级。模板字段{{ .GroupLabels.alertname }}与{{ range .Alerts }}{{ .Annotations.description }}{{ end }}引用告警的组标签与注解send_resolved: true让恢复通知自动发出形成闭环。凭据外置slack_api_url、service_key均以环境变量形式注入避免密钥入库。六、SLO 与错误预算把可靠性变成可量化决策monitor-setup 中 SLO 部分以 TypeScript 的SLOManager演示了目标定义 → 错误预算 → 消费速率 → Prometheus 录制规则的完整链路// slo-manager.ts interface SLO { name: string; target: number; // e.g., 99.9 window: string; // e.g., 30d burnRates: BurnRate[]; } export class SLOManager { private slos: SLO[] [ { name: API Availability, target: 99.9, window: 30d, burnRates: [ { window: 1h, threshold: 14.4, severity: critical }, { window: 6h, threshold: 6, severity: critical }, { window: 1d, threshold: 3, severity: warning }, ], }, ]; generateSLOQueries(): string { return this.slos.map((slo) this.generateSLOQuery(slo)).join(\n\n); } private generateSLOQuery(slo: SLO): string { const errorBudget 1 - slo.target / 100; return # ${slo.name} SLO - record: slo:${this.sanitizeName(slo.name)}:error_budget expr: ${errorBudget} - record: slo:${this.sanitizeName(slo.name)}:consumed_error_budget expr: | 1 - (sum(rate(successful_requests[${slo.window}])) / sum(rate(total_requests[${slo.window}]))) ; } }这里体现了 SLO 工程的两个核心概念错误预算errorBudget 1 - target/100对 99.9% 目标即 0.1%——对应每月约 43.2 分钟的允许不可用时间slo-implementation 技能的 目标换算表 给出了 99%99.99% 的月度/年度折算。多窗口烧毁速率burn rate通过短窗口高倍数 长窗口低倍数组合告警兼顾快速响应与误报抑制——1 小时窗口烧 14.4x2 小时内烧掉 2% 预算需立刻分页、6 小时窗口烧 6x、1 天窗口烧 3x。slo-implement 命令plugins/observability-monitoring/commands/slo-implement.md给出了可直接落地的 Prometheus 录制规则与多窗口烧毁告警例如# 多窗口多烧毁速率告警节选 - alert: ErrorBudgetFastBurn expr: | ( service:error_budget_burn_rate_5m{serviceapi} 14.4 AND service:error_budget_burn_rate_1h{serviceapi} 14.4 ) for: 2m labels: severity: critical team: platform annotations: summary: Fast error budget burn for {{ $labels.service }} description: | Service {{ $labels.service }} is burning error budget at 14.4x rate. 这将使月度预算在 1 小时内消耗 2%。AND连接两个窗口的意义在于只有短窗口与长窗口同时超阈值才触发既不会漏掉突发的预算流失也不会被 5 分钟的毛刺误触发。SLO 落地的完整方法论SLI 定义 → 目标选择 → 预算计算 → 记录规则 → 仪表盘 → 月度报告 → 发布决策可继续参考 slo-implement.md 与 slo-implementation 技能其典型 SLI 定义如下Availability 与 Latency 的 PromQL 形态# 可用性 SLI非 5xx 请求 / 总请求28 天窗口 sum(rate(http_requests_total{status!~5..}[28d])) / sum(rate(http_requests_total[28d])) # 时延 SLI500ms 请求 / 总请求 sum(rate(http_request_duration_seconds_bucket{le0.5}[28d])) / sum(rate(http_request_duration_seconds_count[28d]))七、基础设施即代码Terraform 一键拉起监控栈monitor-setup 最后给出用 Terraform 将 Prometheus、Grafana、Alertmanager 模块化托管的配置# monitoring.tf module prometheus { source ./modules/prometheus namespace monitoring storage_size 100Gi retention_days 30 external_labels { cluster var.cluster_name region var.region } } module grafana { source ./modules/grafana namespace monitoring admin_password var.grafana_admin_password datasources [ { name Prometheus type prometheus url http://prometheus:9090 } ] } module alertmanager { source ./modules/alertmanager namespace monitoring config templatefile(${path.module}/alertmanager.yml, { slack_webhook var.slack_webhook pagerduty_key var.pagerduty_service_key }) }模块化三个 module 各自封装组件细节source指向本地模块目录./modules/*namespace统一收敛到monitoring。参数化存储大小100Gi与保留期30 天作为显式参数与第五节 Helm 部署时的--set ...retention30d一一对应external_labels从变量注入保证跨环境test/staging/prod复用同一套配置。模板注入Alertmanager 配置用templatefile()把slack_webhook、pagerduty_key两个变量渲染进alertmanager.yml配合var.*变量或环境变量实现配置与密钥分离。这一节与 grafana-dashboards 技能中的Dashboard as Codegrafana_dashboard资源 grafana_folder互相补充监控栈的部署、数据源、仪表盘、告警全部进入版本库可评审、可回滚。observability-engineer Agent 的能力描述中也明确包含Observability as Code与GitOps workflows for dashboard and alert management见 plugins/observability-monitoring/agents/observability-engineer.md。交付物清单一次命令的完整产出monitor-setup 命令在交付时要求覆盖以下 8 类产出这也是衡量监控体系是否可行动的检查清单Infrastructure Assessment基础设施评估现有监控能力盘点已有哪些采集、盲区在哪、覆盖度如何。Monitoring Architecture监控架构完整监控栈设计明确指标/日志/追踪三条数据通路与各组件职责。Implementation Plan实施计划分步部署指南含依赖顺序、验证节点与回滚预案。Metric Definitions指标目录全面指标清单命名、类型、标签、采集来源。Dashboard Templates仪表盘模板可直接导入 Grafana 的仪表盘 JSON见第二节。Alert Runbooks告警运行手册逐条告警的响应流程、分级与升级路径。SLO DefinitionsSLO 定义服务水平目标与错误预算见第六节。Integration Guide接入指南服务侧埋点与接入说明见第一、三、四节。整套方案的核心目标写在本命令末尾构建一个能提供可行动洞察actionable insights、降低 MTTR、并主动发现问题的监控系统——即不是指标堆砌而是每个面板、每条告警、每个 SLO 都指向明确的运维决策。相关资源导航命令本体plugins/observability-monitoring/commands/monitor-setup.mdSLO 专项命令plugins/observability-monitoring/commands/slo-implement.mdPrometheus 技能SKILL.md 与 细节参考Grafana 技能plugins/observability-monitoring/skills/grafana-dashboards/SKILL.md分布式追踪技能SKILL.md 与 细节参考SLO 技能plugins/observability-monitoring/skills/slo-implementation/SKILL.md插件 Agentplugins/observability-monitoring/agents/observability-engineer.md命令调用方式与组合工作流docs/usage.md、docs/architecture.md【免费下载链接】agentsMulti-harness agentic plugin marketplace for Claude Code, Codex, Cursor, OpenCode, GitHub Copilot, and Google Antigravity项目地址: https://gitcode.com/GitHub_Trending/agents24/agents创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考