下午突发奇想,天天企业微信发送告警,但是没有统计过告警都收到过哪些,都是哪些级别,持续的时间又是多长,下午在官方文档找到一个项目alertsnitch
。可以对我们的Alertmanager发送的告警进行持久化,实际上就是一个webhook接收器,Alertmanager的数据采集同步写入到Mysql中为以后的数据分析
效果图下
由于官方Alertmanager 本身不支持的通知机制,所以我们需要webhook 接收器进行集成,将数据发送给后端Mysql或者是Postgres
官方gitlab地址:https://gitlab.com/yakshaving.art/alertsnitch
数据我这里存储在MySQL中,我们需要安装一套Mysql配置
创建持久化目录
mkdir -p /data/mysql/{conf,data,logs}
创建自定义my.cnf
cat >/data/mysql/conf/my.cnf<<EOF
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
secure-file-priv= NULL
lower_case_table_names = 1
EOF
lower_case_table_names
需要开启,不区分大小写
启动mysql容器
docker run -p 3306:3306 --name mysql \
-v /data/mysql/conf:/etc/mysql/conf.d \
-v /data/mysql/logs:/logs \
-v /data/mysql/data:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=abcdocker -d \
--restart=always mysql
#这里我临时安装一套mysql,可以根据实际场景自行部署
创建用户
[root@prometheus alert]# docker exec -it mysql bash
root@b7b076e1c2a9:/# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.27 MySQL Community Server - GPL
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create user 'i4t'@'%' identified by 'daskjdlajdas@3';
Query OK, 0 rows affected (0.02 sec)
mysql> grant all privileges on *.* to 'i4t'@'%' ;
Query OK, 0 rows affected (0.01 sec)
我们在创建个库,后续使用i4t
将数据写进库里
mysql> create database alert;
Query OK, 1 row affected (0.01 sec)
接下来在alert库中写入一些表结构数据
#进入到业务库中
USE alert
DROP PROCEDURE IF EXISTS bootstrap;
DELIMITER //
CREATE PROCEDURE bootstrap()
BEGIN
SET @exists := (SELECT 1 FROM information_schema.tables I WHERE I.table_name = "Model" AND I.table_schema = database());
IF @exists IS NULL THEN
CREATE TABLE `Model` (
`ID` enum('1') NOT NULL,
`version` VARCHAR(20) NOT NULL,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO `Model` (`version`) VALUES ("0.0.1");
ELSE
SIGNAL SQLSTATE '42000' SET MESSAGE_TEXT='Model Table Exists, quitting...';
END IF;
END;
//
DELIMITER ;
-- Execute the procedure
CALL bootstrap();
-- Drop the procedure
DROP PROCEDURE bootstrap;
-- Create the rest of the tables
CREATE TABLE `AlertGroup` (
`ID` INT NOT NULL AUTO_INCREMENT,
`time` TIMESTAMP NOT NULL,
`receiver` VARCHAR(100) NOT NULL,
`status` VARCHAR(50) NOT NULL,
`externalURL` TEXT NOT NULL,
`groupKey` VARCHAR(255) NOT NULL,
KEY `idx_time` (`time`) USING BTREE,
KEY `idx_status_ts` (`status`, `time`) USING BTREE,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `GroupLabel` (
`ID` INT NOT NULL AUTO_INCREMENT,
`AlertGroupID` INT NOT NULL,
`GroupLabel` VARCHAR(100) NOT NULL,
`Value` VARCHAR(1000) NOT NULL,
FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `CommonLabel` (
`ID` INT NOT NULL AUTO_INCREMENT,
`AlertGroupID` INT NOT NULL,
`Label` VARCHAR(100) NOT NULL,
`Value` VARCHAR(1000) NOT NULL,
FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `CommonAnnotation` (
`ID` INT NOT NULL AUTO_INCREMENT,
`AlertGroupID` INT NOT NULL,
`Annotation` VARCHAR(100) NOT NULL,
`Value` VARCHAR(1000) NOT NULL,
FOREIGN KEY (AlertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `Alert` (
`ID` INT NOT NULL AUTO_INCREMENT,
`alertGroupID` INT NOT NULL,
`status` VARCHAR(50) NOT NULL,
`startsAt` DATETIME NOT NULL,
`endsAt` DATETIME DEFAULT NULL,
`generatorURL` TEXT NOT NULL,
FOREIGN KEY (alertGroupID) REFERENCES AlertGroup (ID) ON DELETE CASCADE,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `AlertLabel` (
`ID` INT NOT NULL AUTO_INCREMENT,
`AlertID` INT NOT NULL,
`Label` VARCHAR(100) NOT NULL,
`Value` VARCHAR(1000) NOT NULL,
FOREIGN KEY (AlertID) REFERENCES Alert (ID) ON DELETE CASCADE,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `AlertAnnotation` (
`ID` INT NOT NULL AUTO_INCREMENT,
`AlertID` INT NOT NULL,
`Annotation` VARCHAR(100) NOT NULL,
`Value` VARCHAR(1000) NOT NULL,
FOREIGN KEY (AlertID) REFERENCES Alert (ID) ON DELETE CASCADE,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
在修改Model源版本
ALTER TABLE Alert
ADD `fingerprint` TEXT NOT NULL
;
UPDATE `Model` SET `version`="0.1.0";
启动项目
#首先设置环境变量
export ALERTSNITCH_BACKEND="mysql"
export ALERTSNITCH_DSN=DB_USER:DB_PASSWORD@(DB_IP:DB_PORT)/DB_NAME"
#根据实际情况修改
启动Docker 容器
docker run -itd \
-p 9567:9567 \
--name alertsnitch \
-e ALERTSNITCH_DSN \
-e ALERTSNITCH_BACKEND \
registry.gitlab.com/yakshaving.art/alertsnitch
接下来配置Alertmanager
#receivers告警信息如下
receivers:
- name: alertsnitch
webhook_configs:
- url: http://<alertsnitch-host-or-ip>:9567/webhook
Alert route如下
route:
routes:
- receiver: alertsnitch
continue: true
完整配置Alertmanager告警配置如下
[root@prometheus alertmanager]# cat config.yml
global:
resolve_timeout: 5m
http_config:
follow_redirects: true
smtp_hello: localhost
smtp_require_tls: true
pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
opsgenie_api_url: 'https://api.opsgenie.com/'
wechat_api_url: 'https://qyapi.weixin.qq.com/cgi-bin/ '
wechat_api_corp_id: wwxxxx #企业id
victorops_api_url: 'https://alert.victorops.com/integrations/generic/20131114/alert/'
route:
receiver: "null"
group_wait: 30s
group_interval: 3m
repeat_interval: 3m
routes:
- receiver: abcdocker
continue: true
- receiver: alertsnitch
continue: false
receivers:
- name: abcdocker
wechat_configs:
- send_resolved: true
http_config:
follow_redirects: true
api_secret: xxxW-rsXEtY_oFTvLk # 申请企业微信应用后生成的密码
corp_id: wwcxxx50d
message: '{{ template "wechat.default.message" . }}'
api_url: https://qyapi.weixin.qq.com/cgi-bin/
to_user: abcdocker #发送到某一用户也可以 @all 就是群组全员发送
to_party: '{{ template "wechat.default.to_party" . }}'
to_tag: '{{ template "wechat.default.to_tag" . }}'
agent_id: "100004" #申请企业微信应用id
message_type: text
- name: "null"
- name: alertsnitch
webhook_configs:
- send_resolved: true
http_config:
follow_redirects: true
url: http://10.0.24.13:9567/webhook
max_alerts: 0
templates:
- /etc/alertmanager/template/*.tmpl #告警模板路径
重启Alertmanager服务
[root@prometheus alertmanager]# docker restart 9780c1d1d924
9780c1d1d924
Alertmanager 启动成功后,我们需要去Grafana配置Mysql数据源
因为数据都存储在Mysql中,所以我们grafana的数据源来自Mysql
搜索Mysql数据源
根据刚刚我们创建的用户信息填写
检测是否正常通信
接下来导入模板,ID15833
导入即可
在Mysql中已经可以看到日志了
Grafana 效果图如下