Minio是一个开源的对象存储服务器，它兼容Amazon S3的API，被广泛用于构建分布式存储系统和云原生应用程序。

Minio支持集成prometheus，用以监控CPU、硬盘、网络等数据。

Minio集群安装

编辑/etc/hosts文件，添加节点host信息，这里我们部署5个节点。

192.168.3.33   minio1
192.168.3.34   minio2
192.168.3.35   minio3
192.168.3.36   minio4
192.168.3.37   minio5

docker-compose

  minio:
    image: minio/minio
    container_name: minio
    restart: always
    network_mode: "host"
    expose:
      - '9000'
      - '9001'
    volumes:
      - /data/minio/data:/data
      - /etc/localtime:/etc/localtime
      - /etc/timezone:/etc/timezone
    environment:
      MINIO_ROOT_USER: 用户名
      MINIO_ROOT_PASSWORD: 密码
      MINIO_PROMETHEUS_AUTH_TYPE: public
    command: server --console-address ":9001" http://minio{1...5}/data

参数说明

  • MINIO_ROOT_USER，MINIO_ROOT_PASSWORD 用户名密码用于登录9001端口的后台管理。

  • MINIO_PROMETHEUS_AUTH_TYPE: public 允许不需要认证访问metric数据。

启动Minio集群

在每个节点上运行

docker-compose up -d

注意：5个节点间的9000端口要求互通。

测试访问metrics接口

http://192.168.3.33:9000/minio/prometheus/metrics

输出内容如下：

# HELP bucket_objects_count Total number of objects in a bucket
# TYPE bucket_objects_count gauge
bucket_objects_count{bucket="***"} 180
bucket_objects_count{bucket="***"} 318
# HELP bucket_objects_histogram Total number of objects of different sizes in a bucket
# TYPE bucket_objects_histogram gauge
...
# HELP s3_tx_bytes_total Total number of s3 bytes sent by current MinIO server instance
# TYPE s3_tx_bytes_total counter
s3_tx_bytes_total 4.576492e+06
# HELP self_heal_time_since_last_activity Time elapsed (in nano seconds) since last self healing activity. This is set to -1 until initial self heal activity
# TYPE self_heal_time_since_last_activity gauge
self_heal_time_since_last_activity 0

监控方式一

Prometheus配置

修改prometheus.yml，增加job_name

  - job_name: minio
    metrics_path: /minio/prometheus/metrics
    scrape_interval: 10s
    scheme: http
    static_configs:
      - targets: ['192.168.3.33:9000','192.168.3.34:9000','192.168.3.35:9000','192.168.3.36:9000','192.168.3.37:9000']

重载配置

curl -X POST localhost:9090/-/reload -u 用户名:密码

注：这里的用户名密码是prometheus的，不要搞混。

访问你的https://prometheus.xxx.com/targets?search=#pool-minio，确保状态UP

2024-01-17T13:59:32.123726265-hjgz.pngGrafana模板

https://grafana.com/grafana/dashboards/12063-minio/

修改S3接口总请求控件

sum(s3_requests_total{instance="172.16.62.150:9000",job="minio-metrics"}) by (api)
修改为
sum(s3_requests_total{instance="$metrics_port",job="$job"}) by (api)

效果图如下

2024-01-17T14:05:23.108741680-fmso.png监控方式二

Prometheus配置

  - job_name: minio-cluster
    metrics_path: /minio/v2/metrics/cluster
    scrape_interval: 10s
    scheme: http
    static_configs:
      - targets: ['192.168.3.33:9000','192.168.3.34:9000','192.168.3.35:9000','192.168.3.36:9000','192.168.3.37:9000']

Grafana模板

https://grafana.com/grafana/dashboards/13502-minio-dashboard/

告警规则

https://samber.github.io/awesome-prometheus-alerts/rules#minio

已知问题

访问控制台Monitoring-Metrics跳转到登录界面。

在docker-compose.yml文件中新增环境变量：

      MINIO_PROMETHEUS_URL: "http://prometheus IP:9090"
      MINIO_PROMETHEUS_JOB_ID: " minio-cluster"

但是猫子的prometheus设置了basic_auth，在官网和网上均未查到配置方法。