多渠道报警
背景
统一监控报警平台
使用Grafana汇聚了,Prometheus,ELK,CloudWatch监控信息.
简单理解 ELK,AWS均可以实现短信和电话报警
需求
1.Grafana支持短信渠道报警和电话渠道报警.
2.对不同微服务的开发团队发送报警信息.
要求
1.阿里云短信&电话服务API
2.Jenkins-plugin:Generic Webhook Trigger Plugin
逻辑
-
阿里云
- 下载阿里云短信服务SDK
- 创建短信服务ACCESS_KEY_ID
-
Grafana
- 产生报警信息
- 将报警信息发送给Jenkins的webhook接口
-
Jenkins
- 创建JOB,执行python脚本
- 设置JOB的webhook地址
- 创建电话号码变量
上图说明
首先看下阿里云短信接口的python demo
# -*- coding: utf-8 -*-
import sys
from aliyunsdkdysmsapi.request.v20170525 import SendSmsRequest
from aliyunsdkdysmsapi.request.v20170525 import QuerySendDetailsRequest
from aliyunsdkcore.client import AcsClient
import uuid
from aliyunsdkcore.profile import region_provider
from aliyunsdkcore.http import method_type as MT
from aliyunsdkcore.http import format_type as FT
import const
import json
"""
短信业务调用接口示例,版本号:v20170525
Created on 2017-06-12
"""
try:
reload(sys)
sys.setdefaultencoding('utf8')
except NameError:
pass
except Exception as err:
raise err
# 注意:不要更改
REGION = "cn-hangzhou"
PRODUCT_NAME = "Dysmsapi"
DOMAIN = "dysmsapi.aliyuncs.com"
#Jenkins webhook 变量
PhoneNumbers= sys.argv[1].split(",")
Contents= sys.argv[2]
State= sys.argv[3]
print "PhoneNumbers:%s" % PhoneNumbers
print "Contents: %s" % Contents
acs_client = AcsClient(const.ACCESS_KEY_ID, const.ACCESS_KEY_SECRET, REGION)
region_provider.add_endpoint(PRODUCT_NAME, REGION, DOMAIN)
def send_sms(business_id, phone_numbers, sign_name, template_code, template_param=None):
smsRequest = SendSmsRequest.SendSmsRequest()
# 申请的短信模板编码,必填
smsRequest.set_TemplateCode(template_code)
# 短信模板变量参数
if template_param is not None:
smsRequest.set_TemplateParam(template_param)
# 设置业务请求流水号,必填。
smsRequest.set_OutId(business_id)
# 短信签名
smsRequest.set_SignName(sign_name)
# 数据提交方式
# smsRequest.set_method(MT.POST)
# 数据提交格式
# smsRequest.set_accept_format(FT.JSON)
# 短信发送的号码列表,必填。
smsRequest.set_PhoneNumbers(phone_numbers)
# 调用短信发送接口,返回json
smsResponse = acs_client.do_action_with_exception(smsRequest)
# TODO 业务处理
return smsResponse
if __name__ == '__main__':
__business_id = uuid.uuid1()
#print(__business_id)
params = {"name":State,"contents":Contents}
params = json.dumps(params)
print params
for PhoneNumber in PhoneNumbers:
print(send_sms(__business_id, PhoneNumber, "玉符", "SMS_142145090", params))
再看下阿里云电话接口的python demo
# -*- coding: utf-8 -*-
from aliyunsdkdyvmsapi.request.v20170525 import SingleCallByTtsRequest
from aliyunsdkdyvmsapi.request.v20170525 import SingleCallByVoiceRequest
from aliyunsdkcore.client import AcsClient
import uuid
import json
import sys
import re
"""
语音业务调用接口示例,版本号:v20170525
Created on 2017-06-12
安装依赖库
wget http://ytx-sdk.oss-cn-shanghai.aliyuncs.com/dyvms_python.zip
unzip dyvms_python.zip
sudo python dyvms_python/api_demo/aliyun-python-sdk-core/setup.py install
sudo python dyvms_python/api_demo/aliyun-python-sdk-dyvmsapi/setup.py install
sudo python dyvms_python/api_sdk/aliyun-python-sdk-core/setup.py install
sudo python dyvms_python/api_sdk/aliyun-python-sdk-dyvmsapi/setup.py install
"""
#暂时不支持多region,默认配置杭州
REGION = "cn-hangzhou"
# ACCESS_KEY_ID/ACCESS_KEY_SECRET 根据实际申请的账号信息进行替换
ACCESS_KEY_ID = "xxxxxxx"
ACCESS_KEY_SECRET = "xxxxxx"
#初始化AcsClient
acs_client = AcsClient(ACCESS_KEY_ID, ACCESS_KEY_SECRET, REGION)
def tts_call(business_id, called_number, called_show_number, tts_code, tts_param=None):
ttsRequest = SingleCallByTtsRequest.SingleCallByTtsRequest()
# 申请的语音通知tts模板编码,必填
ttsRequest.set_TtsCode(tts_code)
# 设置业务请求流水号,必填。后端服务基于此标识区分是否重复请求的判断
ttsRequest.set_OutId(business_id)
# 语音通知的被叫号码,必填。
ttsRequest.set_CalledNumber(called_number)
# 语音通知显示号码,必填。
ttsRequest.set_CalledShowNumber(called_show_number)
# tts模板变量参数
if tts_param is not None:
ttsRequest.set_TtsParam(tts_param)
ttsResponse = acs_client.do_action_with_exception(ttsRequest)
return ttsResponse
__business_id = uuid.uuid1()
print __business_id
#模板中不存在变量的情况下为{}
PhoneNumbers = sys.argv[1].split(",")
Contents = re.sub('[\[\]]', '', sys.argv[2])
State = sys.argv[3]
Appname = sys.argv[4].split("-")[1]
Level = "高级别"
Value = "-"
params = {"level": Level, "Appname": Appname, "Contents": Contents}
params = json.dumps(params)
print Appname, Contents
print "State:%s" % State
if State == "alerting":
for PhoneNumber in PhoneNumbers:
print "正在呼叫%s" % PhoneNumber
print tts_call(__business_id, PhoneNumber,
"073182705718", "TTS_148381535", params)
安装Jenkins插件
访问jenkins页面 https://jenkins.$demo.com/pluginManager/available
搜索并安装插件 Generic Webhook Trigger Plugin
创建Jenkins Job (下图是几处关键设置)
短信
电话
Jenkins Job创建完成
创建Grafana Notification channels
Url: https://jenkins.$dome.com/generic-webhook-trigger/invoke?token=sms (此处的sms是jenkins中的token)
完成
在Grafana触发报警后会通过这个Channel 发送一个POST请求到Jenkins.
Jenkins会执行对应的JOB
job执行结果