Monitoring Servers with Collectd, InfluxDB & Grafana
In this tutorial we are going to setup a complete monitoring stack which has following components
Collect performance metrics from machines – Collectd
Store it in a datastore – Influxdb
Visualise timeseries data as graphs – Grafana
We will go through the setup, design and role of individual components and then jump into configuring individual components and linking them together
Collectd
CollectD is a performance monitoring daemon which can collect metrics and send it to various datastores, it support Graphite,Influxdb, statsd … It is built in “C” which makes it performant and portable across devices.
InfluxDB
InfluxDB (Docs here) is a time series database – the kind of databases which are made for storing and retrieving time series data. Imaging collecting data from a machine every 10 seconds and then using various mathematical functions to chart and determine trends over time and functions such as min, max, median over X hours et. There are other timeseries databases in the market, like Graphite, Prometheus ..what makes influxdb better is a superior query language
Grafana
Grafana is a dashboarding tool for building visualisations on top of data. It works well with time series data and support all major timeseries databases like graphite, prometheus , Influxdb ..
Setup
We have three machines with one of the components describer above installed on each , we can have all components on same machine also. But to simplify things we will setup three machines. Collectd application collect the system metrics and send it to Influxdb server. Grafana is running on third machine, which fetch data from influxdb and plot the graphs
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | Machine1 Hostname : collectd.myhost.com IP : 192.168.2.16 Machine2 Hostname : influxdb.myhost.com IP : 192.168.2.17 Machine2 Hostname : grafana.myhost.com IP : 192.168.2.17 |
How to install Collectd?
On Centos we can install collect application using yum
| 1 2 3 | yum install collectd |
Once installed , we can start it using command
| 1 2 3 | systemctl start collectd |
Or on Centos6
| 1 2 3 | service collectd start |
Configuring collect to send metrics to Influxdb
For measuring anything on system – we need to use plugins. For plugins there are two things you need to define – first load the needed plugin and secondly define appropriate settings for them. In following plugins there are default settings built in and simply loading them will start those measurements for us
| 1 2 3 4 5 6 7 | LoadPlugin cpu LoadPlugin load LoadPlugin memory LoadPlugin swap LoadPlugin battery |
Then we will Load and configure network Plugin which is used to send metrics to influxdb
| 1 2 3 4 5 6 | LoadPlugin network Server “192.168.2.17” “25826” |
In the configuration we mentioned the IP and Port number for the influx db host
we will restart collectd after making the changes
Installing and configuring influxDB
You can install Influxdb using yum
| 1 2 3 | yum install influxdb |
Start influxdb using
| 1 2 3 | service influxdb start |
For InfluxDB there are quite a few configuration settings which we are using from template
The configuration file will look like this
| 1 2 3 4 5 6 7 | [collectd] enabled = true bind–address = “:25826” database = “collectd” typesdb = “/usr/share/collectd/types.db |
Installing Grafana
You can install Grafana using Yum directly.
| 1 2 3 | yum install https://grafanarel.s3.amazonaws.com/builds/grafana-4.0.2-1481203731.x86_64.rpm |
You can start Grafana by running:
| 1 2 3 | service grafana–server start |
This will start the grafana-server process as the grafana user, which is created during package installation. The default HTTP port is 3000, and default user and group is admin.
To configure the Grafana server to start at boot time:
| 1 2 3 | /sbin/chkconfig —add grafana–server |
Start the server (via systemd)
| 1 2 3 4 5 | $ systemctl daemon–reload $ systemctl start grafana–server $ systemctl status grafana–server |
Enable the systemd service to start at boot
| 1 2 3 | systemctl enable grafana–server.service |
Once started you can access grafana by launching following url in browser
| 1 2 3 4 5 | http://192.168.2.17:3000 username : admin password : admin |
Linking all the components.
First we will verify if influxdb is getting data from collectd
On machine2, connect to influxdb server using the client
| 1 2 3 | influx –host 192.168.2.19 –port 8086 |
Use the following commands inside influx console to see the time series data
| 1 2 3 4 5 6 | use collectd; show series; |
Sample output is below
If you can see the SERIES data from Machine1, we can confirm Influxdb is getting “timeseries” data from InfluxDB . Now we can go to Grafana and add influxdb as a new datasource. I have created a video on this explaining how to add Influxdb datasource in Grafana and display graphs