使用 iKuai Exporter 监控爱快的网络情况
爱快软路由有非常完善的监控功能,但因为是爱快自己的体系,很不灵活,只能满足基本的使用,没法在 Prometheus 里使用,想要做些告警的功能就更不行了。
再加上最近跑网心云的虚机挂了几天才发现,搞一套内网的监控告警体系迫在眉睫了。对于家庭内网的监控告警,有几个需求需要满足:
- 能统计每天设备的流量
- 能对公网在线状态进行监控告警
- 能对设备的在线状态进行监控告警
部署 Exporter
爱快没有提供 Prometheus 的 Exporter,所以我根据爱快的接口开发了一个 Exporter,代码在:https://github.com/jakeslee/ikuai-exporter 。
已经打包成 Docker 镜像,只需要按以下命令就可以获取到:
1 | docker pull jakes/ikuai-exporter:latest |
可以通过给容器传递环境变量对 Exporter 进行配置,目前有以下参数:
变量名 | 说明 |
---|---|
IK_URL | 爱快地址 |
IK_USER | 爱快登录用户 |
IK_PWD | 爱快登录密码 |
使用命令部署 exporter:
1 | docker run -d -e IK_URL=http://10.10.1.253 -e IK_USER=test -e IK_PWD=test123 \ |
配置 Prometheus
在 Prometheus 的配置文件 prometheus.yml
中增加如下配置:
1 | scrape_configs: |
上面的配置中,10.10.1.253
是爱快的路由器 IP 地址,10.10.1.202:9222
是 Exporter 容器的地址。
默认 Prometheus 采集到的数据是被归到 exporter 的下面,但我们 exporter 和爱快是分开的,这样会导致显示的数据来源错误。
这样配置就可以让指标数据正确从 10.10.1.202:9222
采集到,并正确显示 instance
是 10.10.1.253
。
iKuai Exporter 暴露的指标是以 ikuai 开头的,采集配置成功后就可以在 Prometheus 查询到。
配置 Grafana
配置监控面板
我根据需要配置好了一个 Grafana 的面板,效果如图。
下面是我配置好的监控面板,可以导入到 Grafana 后再根据需要进行微调。
1 | {"annotations":{"list":[{"builtIn":1,"datasource":{"type":"grafana","uid":"-- Grafana --"},"enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","target":{"limit":100,"matchAny":false,"tags":[],"type":"dashboard"},"type":"dashboard"}]},"description":"","editable":true,"fiscalYearStartMonth":0,"graphTooltip":1,"id":6,"links":[],"liveNow":false,"panels":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"decimals":0,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":5,"w":3,"x":0,"y":0},"id":2,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"avg(ikuai_cpu_usage_ratio)","legendFormat":"__auto","range":true,"refId":"A"}],"title":"CPU Usage","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"decimals":0,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"percentunit"},"overrides":[]},"gridPos":{"h":5,"w":3,"x":3,"y":0},"id":4,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"ikuai_memory_usage_bytes/ikuai_memory_size_bytes","instant":false,"legendFormat":"__auto","range":true,"refId":"A"}],"title":"Memory Usage","transformations":[],"type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"dtdhms"},"overrides":[]},"gridPos":{"h":5,"w":2,"x":6,"y":0},"id":6,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_uptime{id=\"host\", instance=\"$instance\"}","legendFormat":"__auto","range":true,"refId":"A"}],"title":"Uptime","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"celsius"},"overrides":[]},"gridPos":{"h":5,"w":2,"x":8,"y":0},"id":8,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_cpu_temperature{instance=\"$instance\"}\n","legendFormat":"__auto","range":true,"refId":"A"}],"title":"Temperature","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"gridPos":{"h":5,"w":2,"x":10,"y":0},"id":14,"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"auto","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_conn_count{id=\"host\", instance=\"$instance\"}","legendFormat":"__auto","range":true,"refId":"A"}],"title":"Connection","type":"stat"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"auto","displayMode":"auto","inspect":false,"minWidth":50},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"bytes"},"overrides":[{"matcher":{"id":"byName","options":"上传"},"properties":[{"id":"custom.width","value":98}]},{"matcher":{"id":"byName","options":"IP"},"properties":[{"id":"custom.width","value":104}]},{"matcher":{"id":"byName","options":"下载"},"properties":[{"id":"custom.width","value":112}]}]},"gridPos":{"h":5,"w":6,"x":12,"y":0},"id":17,"options":{"footer":{"fields":"","reducer":["sum"],"show":false},"frameIndex":0,"showHeader":true,"sortBy":[]},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"sort_desc(increase(ikuai_network_send_bytes{id=~\"device/.*\", instance=\"$instance\"}[$__range]) * on(id, instance) group_left(comment, ip_addr) ikuai_device_info)","format":"table","instant":true,"legendFormat":"__auto","range":false,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"increase(ikuai_network_recv_bytes{id=~\"device/.*\", instance=\"$instance\"}[$__range])","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"B"}],"title":"流量统计","transformations":[{"id":"merge","options":{}},{"id":"organize","options":{"excludeByName":{"Time":true,"id":true,"instance":true,"job":true},"indexByName":{"Time":0,"Value #A":6,"Value #B":7,"comment":2,"id":3,"instance":4,"ip_addr":1,"job":5},"renameByName":{"Time":"","Value #A":"上传","Value #B":"下载","comment":"备注","id":"","instance":"","ip_addr":"IP"}}}],"type":"table"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"auto","displayMode":"auto","inspect":false,"minWidth":50},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"bytes"},"overrides":[]},"gridPos":{"h":5,"w":6,"x":18,"y":0},"id":18,"options":{"footer":{"fields":"","reducer":["sum"],"show":false},"showHeader":true},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"sort_desc((increase(ikuai_network_send_bytes{id=~\"iface/.*\", instance=\"$instance\"}[$__range]) and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info)","format":"table","instant":true,"legendFormat":"__auto","range":false,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"(increase(ikuai_network_recv_bytes{id=~\"iface/.*\", instance=\"$instance\"}[$__range]) and on(id) ikuai_uptime > 0)","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"B"}],"title":"公网流量统计","transformations":[{"id":"merge","options":{}},{"id":"organize","options":{"excludeByName":{"Time":true,"id":true,"instance":true,"job":true},"indexByName":{"Time":0,"Value #A":6,"Value #B":7,"comment":2,"id":3,"instance":4,"interface":1,"job":5},"renameByName":{"Time":"","Value #A":"上传","Value #B":"下载","comment":"备注","id":"","instance":"","ip_addr":"IP"}}}],"type":"table"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"thresholds"},"custom":{"align":"auto","displayMode":"auto","inspect":false,"minWidth":50},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byName","options":"upload"},"properties":[{"id":"custom.displayMode","value":"color-text"}]},{"matcher":{"id":"byName","options":"download"},"properties":[{"id":"custom.displayMode","value":"color-text"}]},{"matcher":{"id":"byName","options":"connection"},"properties":[{"id":"unit","value":"none"},{"id":"decimals"}]},{"matcher":{"id":"byName","options":"mac"},"properties":[{"id":"custom.width"}]}]},"gridPos":{"h":11,"w":12,"x":0,"y":5},"id":12,"options":{"footer":{"fields":"","reducer":["sum"],"show":false},"showHeader":true,"sortBy":[{"desc":true,"displayName":"upload"}]},"pluginVersion":"9.2.5","targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"sort_desc(ikuai_network_send_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"}* on (id) group_left(ip_addr, hostname, comment, mac) ikuai_device_info)","format":"table","instant":true,"legendFormat":"__auto","range":false,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"ikuai_network_recv_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, hostname, comment, mac) ikuai_device_info","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"B"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"ikuai_network_conn_count{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left ikuai_device_info","format":"table","hide":false,"instant":true,"legendFormat":"__auto","range":false,"refId":"C"}],"title":"Devices","transformations":[{"id":"merge","options":{}},{"id":"organize","options":{"excludeByName":{"Time":true,"id":true,"instance":true,"job":true},"indexByName":{"Time":0,"Value #A":9,"Value #B":10,"Value #C":8,"comment":4,"hostname":3,"id":1,"instance":5,"ip_addr":2,"job":7,"mac":6},"renameByName":{"Value #A":"upload","Value #B":"download","Value #C":"connection","hostname":"","id":"","instance":"","ip_addr":"ip","job":""}}}],"type":"table"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"auto","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byName","options":"netin"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":5,"w":12,"x":12,"y":5},"id":10,"options":{"legend":{"calcs":["max","mean","last"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"mode":"single","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_send_kbytes_per_second{id=\"host\", instance=\"$instance\"}","legendFormat":"netout","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_recv_kbytes_per_second{id=\"host\", instance=\"$instance\"}","hide":false,"legendFormat":"netin","range":true,"refId":"B"}],"title":"Host Network IO/s","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"left","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineStyle":{"fill":"solid"},"lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byRegexp","options":".*netin"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":6,"w":12,"x":12,"y":10},"id":15,"options":{"legend":{"calcs":["max","last"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","exemplar":false,"expr":"(ikuai_network_send_kbytes_per_second{id=~\"iface/.*\"} and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info","instant":false,"legendFormat":"{{comment}}-netout","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"(ikuai_network_recv_kbytes_per_second{id=~\"iface/.*\"} and on(id) ikuai_uptime > 0) * on (id) group_left(interface, comment) ikuai_iface_info","hide":false,"legendFormat":"{{comment}}-netin","range":true,"refId":"B"}],"type":"timeseries"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"left","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineStyle":{"fill":"solid"},"lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":true,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byRegexp","options":".*recv"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":8,"w":13,"x":0,"y":16},"id":13,"options":{"legend":{"calcs":["max","lastNotNull"],"displayMode":"table","placement":"right","showLegend":true,"sortBy":"Last *","sortDesc":true},"tooltip":{"mode":"single","sort":"desc"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_send_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, comment) ikuai_device_info","legendFormat":"{{ip_addr}}-{{comment}}-send","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_recv_kbytes_per_second{id=~\"device/.*\", instance=\"$instance\"} * on (id) group_left(ip_addr, comment) ikuai_device_info","hide":false,"legendFormat":"{{ip_addr}}-{{comment}}-recv","range":true,"refId":"B"}],"title":"Device Network IO/s","type":"timeseries"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"description":"","fieldConfig":{"defaults":{"color":{"mode":"palette-classic"},"custom":{"axisCenteredZero":false,"axisColorMode":"text","axisLabel":"","axisPlacement":"left","barAlignment":0,"drawStyle":"line","fillOpacity":0,"gradientMode":"none","hideFrom":{"legend":false,"tooltip":false,"viz":false},"lineInterpolation":"linear","lineStyle":{"fill":"solid"},"lineWidth":1,"pointSize":5,"scaleDistribution":{"type":"linear"},"showPoints":"never","spanNulls":false,"stacking":{"group":"A","mode":"none"},"thresholdsStyle":{"mode":"off"}},"decimals":2,"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]},"unit":"Bps"},"overrides":[{"matcher":{"id":"byRegexp","options":".*recv"},"properties":[{"id":"custom.transform","value":"negative-Y"}]}]},"gridPos":{"h":8,"w":11,"x":13,"y":16},"id":11,"options":{"legend":{"calcs":["max","mean","last"],"displayMode":"table","placement":"right","showLegend":true},"tooltip":{"mode":"multi","sort":"none"}},"targets":[{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_send_kbytes_per_second{id=~\"iface/.*\", instance=\"$instance\"} * on (id) group_left(interface, comment) ikuai_iface_info","legendFormat":"{{interface}}-{{comment}}-send","range":true,"refId":"A"},{"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"editorMode":"code","expr":"ikuai_network_recv_kbytes_per_second{id=~\"iface/.*\", instance=\"$instance\"} * on (id) group_left(interface, comment) ikuai_iface_info","hide":false,"legendFormat":"{{interface}}-{{comment}}-recv","range":true,"refId":"B"}],"title":"Interface Network IO/s","type":"timeseries"}],"refresh":"5s","schemaVersion":37,"style":"dark","tags":[],"templating":{"list":[{"current":{"selected":false,"text":"10.10.1.253","value":"10.10.1.253"},"datasource":{"type":"prometheus","uid":"YQF30ui4z"},"definition":"label_values(ikuai_version, instance)","hide":0,"includeAll":false,"multi":false,"name":"instance","options":[],"query":{"query":"label_values(ikuai_version, instance)","refId":"StandardVariableQuery"},"refresh":1,"regex":"","skipUrlSync":false,"sort":0,"type":"query"}]},"time":{"from":"now-6h","to":"now"},"timepicker":{},"timezone":"","title":"爱快网关监控","uid":"IhtQfGZ4k","version":20,"weekStart":""} |
告警配置
这里以创建公网出口监控告警为例。创建 A,B 两个表达式。A 是查询语句,B 是判断语句。
A 的 Example Range
配置为 now-10m to now
,Query
为:
1 | absent(ikuai_iface_info{id="iface/adslct", internet="PPPOE"}) OR on() vector(0) |
B 设置操作为 Math
, Expression 为 $A == 1
。
接着 Alert Condition
选择 B-expression
。既使用 B 的结果作为告警的条件。
Alert evaluation behavior
配置为 1m for 5m
。意思为每 1 分钟计算一次,持续 5 分钟就告警。
Add details for your alert
配置 Summary
为
1 | 公网出口 {{ $labels.id }} 下线超过 5 分钟! |
其它内容根据自己需要配置,配置完上面说明的保存后就可以生效了,最终效果如下:
Grafana 集成的 Alerts 告警城功能基本和 Alertmanager 是一样的,如果熟悉 Alertmanager 的话配置起来会更得心应手。
使用 iKuai Exporter 监控爱快的网络情况
https://blog.imoe.tech/2022/12/25/48-use-ikuai-exporter-to-gather-metrics/