使用 iKuai Exporter 监控爱快的网络情况

爱快软路由有非常完善的监控功能，但因为是爱快自己的体系，很不灵活，只能满足基本的使用，没法在 Prometheus 里使用，想要做些告警的功能就更不行了。

再加上最近跑网心云的虚机挂了几天才发现，搞一套内网的监控告警体系迫在眉睫了。对于家庭内网的监控告警，有几个需求需要满足：

1. 能统计每天设备的流量
2. 能对公网在线状态进行监控告警
3. 能对设备的在线状态进行监控告警

部署 Exporter

爱快没有提供 Prometheus 的 Exporter，所以我根据爱快的接口开发了一个 Exporter，代码在：https://github.com/jakeslee/ikuai-exporter 。

已经打包成 Docker 镜像，只需要按以下命令就可以获取到：

docker pull jakes/ikuai-exporter:latest

可以通过给容器传递环境变量对 Exporter 进行配置，目前有以下参数：

变量名	说明
IK_URL	爱快地址
IK_USER	爱快登录用户
IK_PWD	爱快登录密码

使用命令部署 exporter：

docker run -d -p 9222:9090 -e IK_URL=http://10.10.1.253 -e IK_USER=test -e IK_PWD=test123 \
  jakes/ikuai-exporter:latest

配置 Prometheus

在 Prometheus 的配置文件 prometheus.yml 中增加如下配置：

scrape_configs:
  - job_name: 'ikuai_exporter'
    scrape_interval: 2s
    scrape_timeout: 2s
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
      - target_label: __address__
        replacement: 10.10.1.202:9222  # exporter.
    static_configs:
      - targets:
          - 10.10.1.253

上面的配置中，10.10.1.253 是爱快的路由器 IP 地址，10.10.1.202:9222 是 Exporter 容器的地址。

默认 Prometheus 采集到的数据是被归到 exporter 的下面，但我们 exporter 和爱快是分开的，这样会导致显示的数据来源错误。

这样配置就可以让指标数据正确从 10.10.1.202:9222 采集到，并正确显示 instance 是 10.10.1.253。

iKuai Exporter 暴露的指标是以 ikuai 开头的，采集配置成功后就可以在 Prometheus 查询到。

配置 Grafana

配置监控面板

我根据需要配置好了一个 Grafana 的面板，效果如图。

下面是我配置好的监控面板，可以导入到 Grafana 后再根据需要进行微调。

面板配置[展开] >folded

{"annotations": {"list": [{"builtIn": 1,"datasource": {"type": "grafana","uid": "-- Grafana --"},"enable": true,"hide": true,"iconColor": "rgba(0, 211, 255, 1)","name": "Annotations & Alerts","target": {"limit": 100,"matchAny": false,"tags": [],"type": "dashboard"},"type": "dashboard"}]},"description": "","editable": true,"fiscalYearStartMonth": 0,"graphTooltip": 1,"id": 1,"links": [],"liveNow": false,"panels": [{"datasource": {"type": "prometheus","uid": "dp-AfZzIz"},"fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"decimals": 0,"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "percentunit"},"overrides": []},"gridPos": {"h": 5,"w": 3,"x": 0,"y": 0},"id": 2,"options": {"colorMode": "value","graphMode": "area","justifyMode": "auto","orientation": "auto","reduceOptions": {"calcs": ["lastNotNull"],"fields": "","values": false},"textMode": "auto"},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "avg(ikuai_cpu_usage_ratio{instance=\"$instance\"})","legendFormat": "__auto","range": true,"refId": "A"}],"title": "CPU Usage","type": "stat"},{"datasource": {"type": "prometheus","uid": "dp-AfZzIz"},"fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"decimals": 0,"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "percentunit"},"overrides": []},"gridPos": {"h": 5,"w": 3,"x": 3,"y": 0},"id": 4,"options": {"colorMode": "value","graphMode": "area","justifyMode": "auto","orientation": "auto","reduceOptions": {"calcs": ["lastNotNull"],"fields": "","values": false},"textMode": "auto"},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "ikuai_memory_usage_bytes{instance=\"$instance\"}/ikuai_memory_size_bytes{instance=\"$instance\"}","instant": false,"legendFormat": "__auto","range": true,"refId": "A"}],"title": "Memory Usage","transformations": [],"type": "stat"},{"datasource": {"type": "prometheus","uid": "dp-AfZzIz"},"description": "","fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null}]},"unit": "dtdhms"},"overrides": []},"gridPos": {"h": 5,"w": 2,"x": 6,"y": 0},"id": 6,"options": {"colorMode": "value","graphMode": "area","justifyMode": "auto","orientation": "auto","reduceOptions": {"calcs": ["lastNotNull"],"fields": "","values": false},"textMode": "auto"},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_uptime{id=\"host\", instance=\"$instance\"}","legendFormat": "__auto","range": true,"refId": "A"}],"title": "Uptime","type": "stat"},{"datasource": {"type": "prometheus"},"fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "celsius"},"overrides": []},"gridPos": {"h": 5,"w": 2,"x": 8,"y": 0},"id": 8,"options": {"colorMode": "value","graphMode": "area","justifyMode": "auto","orientation": "auto","reduceOptions": {"calcs": ["lastNotNull"],"fields": "","values": false},"textMode": "auto"},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_cpu_temperature{instance=\"$instance\"}\n","legendFormat": "__auto","range": true,"refId": "A"}],"title": "Temperature","type": "stat"},{"datasource": {"type": "prometheus"},"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]}},"overrides": []},"gridPos": {"h": 5,"w": 2,"x": 10,"y": 0},"id": 14,"options": {"colorMode": "value","graphMode": "area","justifyMode": "auto","orientation": "auto","reduceOptions": {"calcs": ["lastNotNull"],"fields": "","values": false},"textMode": "auto"},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_network_conn_count{id=\"host\", instance=\"$instance\"}","legendFormat": "__auto","range": true,"refId": "A"}],"title": "Connection","type": "stat"},{"datasource": {"type": "prometheus"},"fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"custom": {"align": "auto","cellOptions": {"type": "auto"},"inspect": false,"minWidth": 50},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "bytes"},"overrides": [{"matcher": {"id": "byName","options": "上传"},"properties": [{"id": "custom.width","value": 98}]},{"matcher": {"id": "byName","options": "IP"},"properties": [{"id": "custom.width","value": 104}]},{"matcher": {"id": "byName","options": "下载"},"properties": [{"id": "custom.width","value": 112}]}]},"gridPos": {"h": 5,"w": 6,"x": 12,"y": 0},"id": 17,"options": {"footer": {"countRows": false,"fields": "","reducer": ["sum"],"show": false},"frameIndex": 0,"showHeader": true,"sortBy": []},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "sort_desc(increase(ikuai_network_send_bytes{id=~\"device/.*\", instance=\"$instance\"}[$__range]) * on(id, instance) group_left(comment, ip_addr) ikuai_device_info)","format": "table","instant": true,"legendFormat": "__auto","range": false,"refId": "A"},{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "increase(ikuai_network_recv_bytes{id=~\"device/.*\", instance=\"$instance\"}[$__range])","format": "table","hide": false,"instant": true,"legendFormat": "__auto","range": false,"refId": "B"}],"title": "流量统计","transformations": [{"id": "merge","options": {}},{"id": "organize","options": {"excludeByName": {"Time": true,"id": true,"instance": true,"job": true},"indexByName": {"Time": 0,"Value #A": 6,"Value #B": 7,"comment": 2,"id": 3,"instance": 4,"ip_addr": 1,"job": 5},"renameByName": {"Time": "","Value #A": "上传","Value #B": "下载","comment": "备注","id": "","instance": "","ip_addr": "IP"}}}],"type": "table"},{"datasource": {"type": "prometheus"},"fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"custom": {"align": "auto","cellOptions": {"type": "auto"},"inspect": false,"minWidth": 50},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "bytes"},"overrides": []},"gridPos": {"h": 5,"w": 6,"x": 18,"y": 0},"id": 18,"options": {"footer": {"countRows": false,"fields": "","reducer": ["sum"],"show": false},"showHeader": true},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "sort_desc((increase(ikuai_network_send_bytes{id=~\"iface/.*\", instance=\"$instance\"}[$__range]) and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info)","format": "table","instant": true,"legendFormat": "__auto","range": false,"refId": "A"},{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "(increase(ikuai_network_recv_bytes{id=~\"iface/.*\", instance=\"$instance\"}[$__range]) and on(id) ikuai_uptime > 0)","format": "table","hide": false,"instant": true,"legendFormat": "__auto","range": false,"refId": "B"}],"title": "公网流量统计","transformations": [{"id": "merge","options": {}},{"id": "organize","options": {"excludeByName": {"Time": true,"id": true,"instance": true,"job": true},"indexByName": {"Time": 0,"Value #A": 6,"Value #B": 7,"comment": 2,"id": 3,"instance": 4,"interface": 1,"job": 5},"renameByName": {"Time": "","Value #A": "上传","Value #B": "下载","comment": "备注","id": "","instance": "","ip_addr": "IP"}}}],"type": "table"},{"datasource": {"type": "prometheus","uid": "dp-AfZzIz"},"description": "","fieldConfig": {"defaults": {"color": {"mode": "thresholds"},"custom": {"align": "auto","cellOptions": {"type": "auto"},"inspect": false,"minWidth": 50},"decimals": 2,"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null}]},"unit": "Bps"},"overrides": [{"matcher": {"id": "byName","options": "upload"},"properties": [{"id": "custom.cellOptions","value": {"type": "color-text"}}]},{"matcher": {"id": "byName","options": "download"},"properties": [{"id": "custom.cellOptions","value": {"type": "color-text"}}]},{"matcher": {"id": "byName","options": "connection"},"properties": [{"id": "unit","value": "none"},{"id": "decimals"}]},{"matcher": {"id": "byName","options": "mac"},"properties": [{"id": "custom.width"}]}]},"gridPos": {"h": 11,"w": 12,"x": 0,"y": 5},"id": 12,"options": {"footer": {"countRows": false,"fields": "","reducer": ["sum"],"show": false},"showHeader": true,"sortBy": [{"desc": true,"displayName": "upload"}]},"pluginVersion": "9.4.13","targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "sort_desc(ikuai_network_send_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"}* on (id, instance) group_left(ip_addr, hostname, comment, mac) ikuai_device_info)","format": "table","instant": true,"legendFormat": "__auto","range": false,"refId": "A"},{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "ikuai_network_recv_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, hostname, comment, mac) ikuai_device_info","format": "table","hide": false,"instant": true,"legendFormat": "__auto","range": false,"refId": "B"},{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "ikuai_network_conn_count{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left ikuai_device_info","format": "table","hide": false,"instant": true,"legendFormat": "__auto","range": false,"refId": "C"}],"title": "Devices","transformations": [{"id": "merge","options": {}},{"id": "organize","options": {"excludeByName": {"Time": true,"id": true,"instance": true,"job": true},"indexByName": {"Time": 0,"Value #A": 9,"Value #B": 10,"Value #C": 8,"comment": 4,"hostname": 3,"id": 1,"instance": 5,"ip_addr": 2,"job": 7,"mac": 6},"renameByName": {"Value #A": "upload","Value #B": "download","Value #C": "connection","hostname": "","id": "","instance": "","ip_addr": "ip","job": ""}}}],"type": "table"},{"datasource": {"type": "prometheus"},"description": "","fieldConfig": {"defaults": {"color": {"mode": "palette-classic"},"custom": {"axisCenteredZero": false,"axisColorMode": "text","axisLabel": "","axisPlacement": "auto","barAlignment": 0,"drawStyle": "line","fillOpacity": 0,"gradientMode": "none","hideFrom": {"legend": false,"tooltip": false,"viz": false},"lineInterpolation": "linear","lineWidth": 1,"pointSize": 5,"scaleDistribution": {"type": "linear"},"showPoints": "never","spanNulls": false,"stacking": {"group": "A","mode": "none"},"thresholdsStyle": {"mode": "off"}},"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "Bps"},"overrides": [{"matcher": {"id": "byName","options": "netin"},"properties": [{"id": "custom.transform","value": "negative-Y"}]}]},"gridPos": {"h": 5,"w": 12,"x": 12,"y": 5},"id": 10,"options": {"legend": {"calcs": ["max", "mean", "last"],"displayMode": "table","placement": "right","showLegend": true},"tooltip": {"mode": "single","sort": "none"}},"targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_network_send_kbytes_per_second{id=\"host\", instance=\"$instance\"}","legendFormat": "netout","range": true,"refId": "A"},{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_network_recv_kbytes_per_second{id=\"host\", instance=\"$instance\"}","hide": false,"legendFormat": "netin","range": true,"refId": "B"}],"title": "Host Network IO/s","type": "timeseries"},{"datasource": {"type": "prometheus"},"description": "","fieldConfig": {"defaults": {"color": {"mode": "palette-classic"},"custom": {"axisCenteredZero": false,"axisColorMode": "text","axisLabel": "","axisPlacement": "left","barAlignment": 0,"drawStyle": "line","fillOpacity": 0,"gradientMode": "none","hideFrom": {"legend": false,"tooltip": false,"viz": false},"lineInterpolation": "linear","lineStyle": {"fill": "solid"},"lineWidth": 1,"pointSize": 5,"scaleDistribution": {"type": "linear"},"showPoints": "never","spanNulls": false,"stacking": {"group": "A","mode": "none"},"thresholdsStyle": {"mode": "off"}},"decimals": 2,"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "Bps"},"overrides": [{"matcher": {"id": "byRegexp","options": ".*netin"},"properties": [{"id": "custom.transform","value": "negative-Y"}]}]},"gridPos": {"h": 6,"w": 12,"x": 12,"y": 10},"id": 15,"options": {"legend": {"calcs": ["max", "last"],"displayMode": "table","placement": "right","showLegend": true},"tooltip": {"mode": "multi","sort": "none"}},"targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","exemplar": false,"expr": "(ikuai_network_send_kbytes_per_second{id=~\"iface/.*\"} and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info","instant": false,"legendFormat": "{{comment}}-netout","range": true,"refId": "A"},{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "(ikuai_network_recv_kbytes_per_second{id=~\"iface/.*\"} and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info","hide": false,"legendFormat": "{{comment}}-netin","range": true,"refId": "B"}],"type": "timeseries"},{"datasource": {"type": "prometheus"},"description": "","fieldConfig": {"defaults": {"color": {"mode": "palette-classic"},"custom": {"axisCenteredZero": false,"axisColorMode": "text","axisLabel": "","axisPlacement": "left","barAlignment": 0,"drawStyle": "line","fillOpacity": 0,"gradientMode": "none","hideFrom": {"legend": false,"tooltip": false,"viz": false},"lineInterpolation": "linear","lineStyle": {"fill": "solid"},"lineWidth": 1,"pointSize": 5,"scaleDistribution": {"type": "linear"},"showPoints": "never","spanNulls": true,"stacking": {"group": "A","mode": "none"},"thresholdsStyle": {"mode": "off"}},"decimals": 2,"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null}]},"unit": "Bps"},"overrides": [{"matcher": {"id": "byRegexp","options": ".*recv"},"properties": [{"id": "custom.transform","value": "negative-Y"}]}]},"gridPos": {"h": 8,"w": 13,"x": 0,"y": 16},"id": 13,"options": {"legend": {"calcs": ["max", "lastNotNull"],"displayMode": "table","placement": "right","showLegend": true,"sortBy": "Last *","sortDesc": true},"tooltip": {"mode": "single","sort": "desc"}},"targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_network_send_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, comment) ikuai_device_info","legendFormat": "{{ip_addr}}-{{comment}}-send","range": true,"refId": "A"},{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_network_recv_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, comment) ikuai_device_info","hide": false,"legendFormat": "{{ip_addr}}-{{comment}}-recv","range": true,"refId": "B"}],"title": "Device Network IO/s","type": "timeseries"},{"datasource": {"type": "prometheus"},"description": "","fieldConfig": {"defaults": {"color": {"mode": "palette-classic"},"custom": {"axisCenteredZero": false,"axisColorMode": "text","axisLabel": "","axisPlacement": "left","barAlignment": 0,"drawStyle": "line","fillOpacity": 0,"gradientMode": "none","hideFrom": {"legend": false,"tooltip": false,"viz": false},"lineInterpolation": "linear","lineStyle": {"fill": "solid"},"lineWidth": 1,"pointSize": 5,"scaleDistribution": {"type": "linear"},"showPoints": "never","spanNulls": false,"stacking": {"group": "A","mode": "none"},"thresholdsStyle": {"mode": "off"}},"decimals": 2,"mappings": [],"thresholds": {"mode": "absolute","steps": [{"color": "green","value": null},{"color": "red","value": 80}]},"unit": "Bps"},"overrides": [{"matcher": {"id": "byRegexp","options": ".*recv"},"properties": [{"id": "custom.transform","value": "negative-Y"}]}]},"gridPos": {"h": 8,"w": 11,"x": 13,"y": 16},"id": 11,"options": {"legend": {"calcs": ["max", "mean", "last"],"displayMode": "table","placement": "right","showLegend": true},"tooltip": {"mode": "multi","sort": "none"}},"targets": [{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_network_send_kbytes_per_second{id=~\"iface/.*\", instance=\"$instance\"} * on (id) group_left(interface, comment) ikuai_iface_info","legendFormat": "{{interface}}-{{comment}}-send","range": true,"refId": "A"},{"datasource": {"type": "prometheus"},"editorMode": "code","expr": "ikuai_network_recv_kbytes_per_second{id=~\"iface/.*\", instance=\"$instance\"} * on (id) group_left(interface, comment) ikuai_iface_info","hide": false,"legendFormat": "{{interface}}-{{comment}}-recv","range": true,"refId": "B"}],"title": "Interface Network IO/s","type": "timeseries"}],"refresh": "5s","revision": 1,"schemaVersion": 38,"style": "dark","tags": [],"templating": {"list": [{"current": {"selected": false,"text": "10.0.1.253","value": "10.0.1.253"},"datasource": {"type": "prometheus","uid": "dp-AfZzIz"},"definition": "label_values(ikuai_version, instance)","hide": 0,"includeAll": false,"multi": false,"name": "instance","options": [],"query": {"query": "label_values(ikuai_version, instance)","refId": "StandardVariableQuery"},"refresh": 1,"regex": "","skipUrlSync": false,"sort": 0,"type": "query"}]},"time": {"from": "now-6h","to": "now"},"timepicker": {},"timezone": "","title": "爱快网关监控","uid": "IhtQfGZ4k","version": 4,"weekStart": ""}

告警配置

这里以创建公网出口监控告警为例。创建 A，B 两个表达式。A 是查询语句，B 是判断语句。

A 的 Example Range 配置为 now-10m to now，Query 为：

A

absent(ikuai_iface_info{id="iface/adslct", internet="PPPOE"}) OR on() vector(0)

Options 设置 type 为 Instant。

B 设置操作为 Math, Expression 为 $A == 1。

接着 Alert Condition 选择 B-expression。既使用 B 的结果作为告警的条件。

Alert evaluation behavior 配置为 1m for 5m。意思为每 1 分钟计算一次，持续 5 分钟就告警。

Add details for your alert 配置 Summary 为

Summary

公网出口 {{ $labels.id }} 下线超过 5 分钟！

其它内容根据自己需要配置，配置完上面说明的保存后就可以生效了，最终效果如下：

Grafana 集成的 Alerts 告警城功能基本和 Alertmanager 是一样的，如果熟悉 Alertmanager 的话配置起来会更得心应手。

修订历史

• 2023-09-28：修复 Grafana 面板多实例兼容问题。Issued by @scoryyi
• 2023-08-30：告警设置里 Options 参数补充说明。
• 2023-08-26：修复导入面板后数据源错误问题。Issued by @kun
• 2023-04-27：exporter 启动命令错误。Issued by @cybertech