Loggie's Monitoring and Alarming¶
Loggie's monitor eventbus is designed in a publish and subscribe mode. Each component sends metrics to a specified topic, which is consumed and processed by an independent listener.
For example, file source
collects some indicators from the collected logs and send them to filesource topic
. After aggregated and calculated by filesource listener
, these indicators will be printed and exposed as Prometheus indicators.
There is a loose coupling relationship between components, topics and listeners. For example, file source
will regularly send the full matching indicators to filewatcher topic
, filewatcher listener
process and expose the indicators.
Monitor Configuration¶
The monitor eventbus is configured in the global system configuration, the example is as follows:
Config
loggie:
monitor:
logger:
period: 30s
enabled: true
listeners:
filesource: ~
filewatcher: ~
reload: ~
queue: ~
sink: ~
http:
enabled: true
port: 9196
logger
controlls the log printing of all metrics indicators. Metrics generated by the configured listeners will be aggregated and printed in Loggie log at fixed intervals (set by period
), which is convenient for backtracking and troubleshooting.
listeners
is used to configure whether the related listener is enabled.
Prometheus metrics are exposed at /metrics
on http.port
by default. You can curl <podIp>:9196/metrics
to view the current metrics.
Core Indicators of Log Collection¶
Currently there are the following listeners:
- filesource: The indicator data about current log collection, such as which files are currently being collected, and what is the collection status?
- filewatcher: Timed full traversal (default 5min) to check all files matching the path, monitor the global collection status, and determine whether there are files that have not been collected in time, etc.
- reload: Number of reloads
- queue: Queue status
- sink: Metrics of sending, such as the number of successes or failures, etc.
Deploy Prometheus and Grafana¶
You can use the existing Prometheus or Grafana. If you need a new deployment, please refer to https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack.
Deploy with Helm:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack -nprometheus --create-namespace
Note
Due to some reasons, some of these k8s.gcr.io
images may not be available for download. You can consider downloading the chart package to replace it and then redeploy it.
After confirming that the Pod is running normally, you can access grafana. The way to access grafana through a proxy can be referred to:
kubectl -nprometheus port-forward --address 0.0.0.0 service/prometheus-grafana 8181:80
Grafana username and password can be viewed in prometheus-grafana sercret. Use base64 -d
.
Added Loggie Prometheus Monitoring¶
In the Kubernetes cluster where Loggie is deployed, create the following ServiceMonitor to allow Prometheus to collect Loggie Agent metrics.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: loggie
release: prometheus
name: loggie-agent
namespace: prometheus
spec:
namespaceSelector:
matchNames:
- loggie
endpoints:
- port: monitor
selector:
matchLabels:
app: loggie
instance: loggie
At the same time, we need to add the json in the install project to Grafana to display Loggie's monitoring console.
Note
The Kubernetes version and the Grafana version may be different, which may lead to incompatible chart display. Modify as you need.
The imported Grafana chart is as shown following: