使用 iKuai Exporter 监控爱快的网络情况

爱快软路由有非常完善的监控功能，但因为是爱快自己的体系，很不灵活，只能满足基本的使用，没法在 Prometheus 里使用，想要做些告警的功能就更不行了。

再加上最近跑网心云的虚机挂了几天才发现，搞一套内网的监控告警体系迫在眉睫了。对于家庭内网的监控告警，有几个需求需要满足：

1. 能统计每天设备的流量
2. 能对公网在线状态进行监控告警
3. 能对设备的在线状态进行监控告警

部署 Exporter

爱快没有提供 Prometheus 的 Exporter，所以我根据爱快的接口开发了一个 Exporter，代码在：https://github.com/jakeslee/ikuai-exporter 。

已经打包成 Docker 镜像，只需要按以下命令就可以获取到：

docker pull jakes/ikuai-exporter:latest

可以通过给容器传递环境变量对 Exporter 进行配置，目前有以下参数：

变量名	说明
IK_URL	爱快地址
IK_USER	爱快登录用户
IK_PWD	爱快登录密码

使用命令部署 exporter：

docker run -d -e IK_URL=http://10.10.1.253 -e IK_USER=test -e IK_PWD=test123 \
  jakes/ikuai-exporter:latest

配置 Prometheus

在 Prometheus 的配置文件 prometheus.yml 中增加如下配置：

scrape_configs:
  - job_name: 'ikuai_exporter'
    scrape_interval: 2s
    scrape_timeout: 2s
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - target_label: __address__
        replacement: 10.10.1.202:9222  # exporter.
    static_configs:
      - targets:
          - 10.10.1.253

上面的配置中，10.10.1.253 是爱快的路由器 IP 地址，10.10.1.202:9222 是 Exporter 容器的地址。

默认 Prometheus 采集到的数据是被归到 exporter 的下面，但我们 exporter 和爱快是分开的，这样会导致显示的数据来源错误。

这样配置就可以让指标数据正确从 10.10.1.202:9222 采集到，并正确显示 instance 是 10.10.1.253。

iKuai Exporter 暴露的指标是以 ikuai 开头的，采集配置成功后就可以在 Prometheus 查询到。

配置 Grafana

配置监控面板

我根据需要配置好了一个 Grafana 的面板，效果如图。

下面是我配置好的监控面板，可以导入到 Grafana 后再根据需要进行微调。

面板配置 >folded

{"annotations":{"list":[{"builtIn":1,"datasource":{"type":"grafana","uid":"-- Grafana --"},"enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","target":{"limit":100,"matchAny":false,"tags":[],"type":"dashboard"},"type":"dashboard"}]},"description":"","editable":true,"fiscalYearStartMonth":0,"graphTooltip":1,"id":6,"links":[],"liveNow":false,"panels":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"decimals":0,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":5,"w":3,"x":0,"y":0},"id":2,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"avg(ikuai_cpu_usage_ratio)","legendFormat":"__auto","range":true,"refId":"A"}],"title":"CPU Usage","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"decimals":0,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":5,"w":3,"x":3,"y":0},"id":4,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"ikuai_memory_usage_bytes/ikuai_memory_size_bytes","instant":false,"legendFormat":"__auto","range":true,"refId":"A"}],"title":"Memory Usage","transformations":[],"type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"dtdhms"},"overrides":[]},"gridPos":{"h":5,"w":2,"x":6,"y":0},"id":6,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_uptime{id=\"host\", instance=\"$instance\"}","legendFormat":"__auto","range":true,"refId":"A"}],"title":"Uptime","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"celsius"},"overrides":[]},"gridPos":{"h":5,"w":2,"x":8,"y":0},"id":8,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_cpu_temperature{instance=\"$instance\"}\n","legendFormat":"__auto","range":true,"refId":"A"}],"title":"Temperature","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":5,"w":2,"x":10,"y":0},"id":14,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_conn_count{id=\"host\", instance=\"$instance\"}","legendFormat":"__auto","range":true,"refId":"A"}],"title":"Connection","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"auto","displayMode":"auto","inspect":false,"minWidth":50},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"bytes"},"overrides":[{"matcher":{"id":"byName","options":"上传"},"properties":[{"id":"custom.width","value":98}]},{"matcher":{"id":"byName","options":"IP"},"properties":[{"id":"custom.width","value":104}]},{"matcher":{"id":"byName","options":"下载"},"properties":[{"id":"custom.width","value":112}]}]},"gridPos":{"h":5,"w":6,"x":12,"y":0},"id":17,"options":{"footer":{"fields":"","reducer":["sum"],"show":false},"frameIndex":0,"showHeader":true,"sortBy":[]},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"sort_desc(increase(ikuai_network_send_bytes{id=~\"device/.*\", instance=\"$instance\"}[$__range]) * on(id, instance) group_left(comment, ip_addr) ikuai_device_info)","format":"table","instant":true,"legendFormat":"__auto","range":false,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"increase(ikuai_network_recv_bytes{id=~\"device/.*\", instance=\"$instance\"}[$__range])","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"B"}],"title":"流量统计","transformations":[{"id":"merge","options":{}},{"id":"organize","options":{"excludeByName":{"Time":true,"id":true,"instance":true,"job":true},"indexByName":{"Time":0,"Value #A":6,"Value #B":7,"comment":2,"id":3,"instance":4,"ip_addr":1,"job":5},"renameByName":{"Time":"","Value #A":"上传","Value #B":"下载","comment":"备注","id":"","instance":"","ip_addr":"IP"}}}],"type":"table"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"auto","displayMode":"auto","inspect":false,"minWidth":50},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"bytes"},"overrides":[]},"gridPos":{"h":5,"w":6,"x":18,"y":0},"id":18,"options":{"footer":{"fields":"","reducer":["sum"],"show":false},"showHeader":true},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"sort_desc((increase(ikuai_network_send_bytes{id=~\"iface/.*\", instance=\"$instance\"}[$__range]) and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info)","format":"table","instant":true,"legendFormat":"__auto","range":false,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"(increase(ikuai_network_recv_bytes{id=~\"iface/.*\", instance=\"$instance\"}[$__range]) and on(id) ikuai_uptime > 0)","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"B"}],"title":"公网流量统计","transformations":[{"id":"merge","options":{}},{"id":"organize","options":{"excludeByName":{"Time":true,"id":true,"instance":true,"job":true},"indexByName":{"Time":0,"Value #A":6,"Value #B":7,"comment":2,"id":3,"instance":4,"interface":1,"job":5},"renameByName":{"Time":"","Value #A":"上传","Value #B":"下载","comment":"备注","id":"","instance":"","ip_addr":"IP"}}}],"type":"table"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"auto","displayMode":"auto","inspect":false,"minWidth":50},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byName","options":"upload"},"properties":[{"id":"custom.displayMode","value":"color-text"}]},{"matcher":{"id":"byName","options":"download"},"properties":[{"id":"custom.displayMode","value":"color-text"}]},{"matcher":{"id":"byName","options":"connection"},"properties":[{"id":"unit","value":"none"},{"id":"decimals"}]},{"matcher":{"id":"byName","options":"mac"},"properties":[{"id":"custom.width"}]}]},"gridPos":{"h":11,"w":12,"x":0,"y":5},"id":12,"options":{"footer":{"fields":"","reducer":["sum"],"show":false},"showHeader":true,"sortBy":[{"desc":true,"displayName":"upload"}]},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"sort_desc(ikuai_network_send_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"}* on (id) group_left(ip_addr, hostname, comment, mac) ikuai_device_info)","format":"table","instant":true,"legendFormat":"__auto","range":false,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"ikuai_network_recv_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, hostname, comment, mac) ikuai_device_info","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"B"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"ikuai_network_conn_count{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left ikuai_device_info","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"C"}],"title":"Devices","transformations":[{"id":"merge","options":{}},{"id":"organize","options":{"excludeByName":{"Time":true,"id":true,"instance":true,"job":true},"indexByName":{"Time":0,"Value #A":9,"Value #B":10,"Value #C":8,"comment":4,"hostname":3,"id":1,"instance":5,"ip_addr":2,"job":7,"mac":6},"renameByName":{"Value #A":"upload","Value #B":"download","Value #C":"connection","hostname":"","id":"","instance":"","ip_addr":"ip","job":""}}}],"type":"table"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byName","options":"netin"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":5,"w":12,"x":12,"y":5},"id":10,"options":{"legend":{"calcs":["max","mean","last"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_send_kbytes_per_second{id=\"host\", instance=\"$instance\"}","legendFormat":"netout","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_recv_kbytes_per_second{id=\"host\", instance=\"$instance\"}","hide":false,"legendFormat":"netin","range":true,"refId":"B"}],"title":"Host Network IO/s","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"left","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineStyle":{"fill":"solid"},"lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byRegexp","options":".*netin"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":6,"w":12,"x":12,"y":10},"id":15,"options":{"legend":{"calcs":["max","last"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"(ikuai_network_send_kbytes_per_second{id=~\"iface/.*\"} and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info","instant":false,"legendFormat":"{{comment}}-netout","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"(ikuai_network_recv_kbytes_per_second{id=~\"iface/.*\"} and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info","hide":false,"legendFormat":"{{comment}}-netin","range":true,"refId":"B"}],"type":"timeseries"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"left","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineStyle":{"fill":"solid"},"lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":true,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byRegexp","options":".*recv"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":8,"w":13,"x":0,"y":16},"id":13,"options":{"legend":{"calcs":["max","lastNotNull"],"displayMode":"table","placement":"right","showLegend":true,"sortBy":"Last *","sortDesc":true},"tooltip":{"mode":"single","sort":"desc"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_send_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, comment) ikuai_device_info","legendFormat":"{{ip_addr}}-{{comment}}-send","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_recv_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, comment) ikuai_device_info","hide":false,"legendFormat":"{{ip_addr}}-{{comment}}-recv","range":true,"refId":"B"}],"title":"Device Network IO/s","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"left","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineStyle":{"fill":"solid"},"lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byRegexp","options":".*recv"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":8,"w":11,"x":13,"y":16},"id":11,"options":{"legend":{"calcs":["max","mean","last"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_send_kbytes_per_second{id=~\"iface/.*\", instance=\"$instance\"} * on (id) group_left(interface, comment) ikuai_iface_info","legendFormat":"{{interface}}-{{comment}}-send","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_recv_kbytes_per_second{id=~\"iface/.*\", instance=\"$instance\"} * on (id) group_left(interface, comment) ikuai_iface_info","hide":false,"legendFormat":"{{interface}}-{{comment}}-recv","range":true,"refId":"B"}],"title":"Interface Network IO/s","type":"timeseries"}],"refresh":"5s","schemaVersion":37,"style":"dark","tags":[],"templating":{"list":[{"current":{"selected":false,"text":"10.10.1.253","value":"10.10.1.253"},"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"definition":"label_values(ikuai_version, instance)","hide":0,"includeAll":false,"multi":false,"name":"instance","options":[],"query":{"query":"label_values(ikuai_version, instance)","refId":"StandardVariableQuery"},"refresh":1,"regex":"","skipUrlSync":false,"sort":0,"type":"query"}]},"time":{"from":"now-6h","to":"now"},"timepicker":{},"timezone":"","title":"爱快网关监控","uid":"IhtQfGZ4k","version":20,"weekStart":""}

告警配置

这里以创建公网出口监控告警为例。创建 A，B 两个表达式。A 是查询语句，B 是判断语句。

A 的 Example Range 配置为 now-10m to now，Query 为：

A

absent(ikuai_iface_info{id="iface/adslct", internet="PPPOE"}) OR on() vector(0)

B 设置操作为 Math, Expression 为 $A == 1。

接着 Alert Condition 选择 B-expression。既使用 B 的结果作为告警的条件。

Alert evaluation behavior 配置为 1m for 5m。意思为每 1 分钟计算一次，持续 5 分钟就告警。

Add details for your alert 配置 Summary 为

Summary

公网出口 {{ $labels.id }} 下线超过 5 分钟！

其它内容根据自己需要配置，配置完上面说明的保存后就可以生效了，最终效果如下：

Grafana 集成的 Alerts 告警城功能基本和 Alertmanager 是一样的，如果熟悉 Alertmanager 的话配置起来会更得心应手。