Prometheus Operator 修改数据存储时间

释放双眼,带上耳机,听听看~!
前面说了prometheus operator持久化的问题,但是还有一个问题很多人都忽略了,那就是prometheus operator数据保留天数,根据官方文档的说明,默认prometheus operator数据存储的时间为1d,这个时候无论你prometheus operator如何进行持久化,都没有作用,因为数据只保留了1天,那么你是无法看到更多天数的数据

Prometheus

关于prometheus operator更多监控可以参考下面的文章

Prometheus Operator 监控k8s组件

默认情况下,prometheus operator已经可以监控我们的集群,但是无法监控kube-controller-manager和kube-scheduler。 这里我们将这2个组件进行监控,并将prometheus和grafana添加traefik。通过ingress进行访问

Prometheus Operator

之前的文章介绍过传统方式安装Prometheus来监控集群,但是这种方式有一定的缺陷虽然完全可以用自定义方式来实现,但是不够灵活,不具有通用性。在自定义监控时还需要不断的更新Prometheus的配置。如果是kubernetes集群完全可以使用一种更原始的方式来实现kubernetes集群监控,即采…

官方文档可以配置的说明

image_1e6io8fmb4os1tp8rmv1oehj2d9.png-69.8kB

图片来自官方文档

如果你的prometheus operator想使用NFS持久化可以看下面的文章,和本次修改prometheus operator默认持久化数据不冲突

Prometheus Operator 持久化存储

因为Prometheus operator默认情况下没有将数据持久化存储,当Pod被删除或者意外重启后,可能会造成数据丢失。

实际上修改prometheus operator时间是通过retention参数进行修改,上面也提示了在prometheus.spec下填写

#接下来进行填写,首先我们先下载operator,并将目录进行分类
wget -P /root/ http://down.i4t.com/abcdocker-prometheus-operator.yaml.zip
cd /root/
unzip abcdocker-prometheus-operator.yaml.zip
mkdir kube-prom
cp -a kube-prometheus-master/manifests/* kube-prom/
cd kube-prom/
mkdir -p node-exporter alertmanager grafana kube-state-metrics prometheus serviceMonitor adapter operator
mv *-serviceMonitor* serviceMonitor/
mv setup operator/
mv grafana-* grafana/
mv kube-state-metrics-* kube-state-metrics/
mv alertmanager-* alertmanager/
mv node-exporter-* node-exporter/
mv prometheus-adapter* adapter/
mv prometheus-* prometheus/
mv 0prometheus-operator-* operator/
mv 00namespace-namespace.yaml operator/


#接下来需要修改prometheus operator的deployment文件
cd /root/kube-prom/prometheus/
vim prometheus-prometheus.yaml
在spec.下面添加
retention: 7d

image_1e6ioheir1latu5k15jo1npiqfm.png-56.6kB

如果已经安装了可以直接修改prometheus-prometheus.yaml 然后通过kubectl apply -f 刷新即可

修改完毕后检查pod运行状态是否正常

[root@YZSJHL82-101 ~]# kubectl  get all -n monitoring 
NAME                                       READY   STATUS    RESTARTS   AGE
pod/alertmanager-main-0                    2/2     Running   0          44h
pod/alertmanager-main-1                    2/2     Running   0          44h
pod/alertmanager-main-2                    2/2     Running   0          44h
pod/grafana-558647b59-tlp8s                1/1     Running   0          44h
pod/kube-state-metrics-5bfc7db74d-8d4jk    4/4     Running   0          44h
pod/node-exporter-jwl9v                    2/2     Running   0          44h
pod/node-exporter-nxn2w                    2/2     Running   0          44h
pod/node-exporter-s6lpr                    2/2     Running   0          44h
pod/prometheus-adapter-57c497c557-svfs8    1/1     Running   0          44h
pod/prometheus-k8s-0                       3/3     Running   1          44h
pod/prometheus-k8s-1                       3/3     Running   1          44h
pod/prometheus-operator-69bd579bf9-tzr5z   1/1     Running   0          44h

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/alertmanager-main       ClusterIP   10.254.149.37            9093/TCP            44h
service/alertmanager-operated   ClusterIP   None                     9093/TCP,6783/TCP   44h
service/grafana                 NodePort    10.254.134.244           3000:20614/TCP      44h
service/kube-state-metrics      ClusterIP   None                     8443/TCP,9443/TCP   44h
service/node-exporter           ClusterIP   None                     9100/TCP            44h
service/prometheus-adapter      ClusterIP   10.254.7.111             443/TCP             44h
service/prometheus-k8s          NodePort    10.254.130.51            9090:13535/TCP      44h
service/prometheus-operated     ClusterIP   None                     9090/TCP            44h
service/prometheus-operator     ClusterIP   None                     8080/TCP            44h

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
daemonset.apps/node-exporter   3         3         3       3            3           beta.kubernetes.io/os=linux   44h

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana               1/1     1            1           44h
deployment.apps/kube-state-metrics    1/1     1            1           44h
deployment.apps/prometheus-adapter    1/1     1            1           44h
deployment.apps/prometheus-operator   1/1     1            1           44h

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-558647b59                1         1         1       44h
replicaset.apps/kube-state-metrics-5bfc7db74d    1         1         1       44h
replicaset.apps/prometheus-adapter-57c497c557    1         1         1       44h
replicaset.apps/prometheus-operator-69bd579bf9   1         1         1       44h

NAME                                 READY   AGE
statefulset.apps/alertmanager-main   3/3     44h
statefulset.apps/prometheus-k8s      2/2     44h

接下来可以访问grafana或者prometheus ui进行检查 (我这里修改完毕后等待2天,检查数据是否正常)

修改前

image_1e6iou4311o1f11db1dck1q6bcs13.png-215.5kB

修改后

image_1e6iov09r8ji8o04rq1h0c1k7g1g.png-281.8kB

人已赞赏
prometheus

Prometheus Operator 持久化存储

2020-3-10 21:00:32

Linux

摘选 《人人都是工程师》前言

2017-2-8 17:26:58

2 条回复 A文章作者 M管理员
  1. Adger

    期待告警prometheus-operator 告警 配置。

个人中心
购物车
优惠劵
今日签到
有新私信 私信列表
搜索