As an alternative to InfluxDB you could store traffic metrics in ClickHouse. This capability is available only for FastNetMon Advanced.
In addition to our legacy / deprecated traffic metrics storage InfluxDB we offer complete support for Clickhouse.
It has many advantages over InfluxDB:
- Unlimited scalability
- Predictable memory usage
- Clustering / redundancy support
- Ability to store not only metrics but traffic too.
We recommend using Clickhouse for metrics for any network which exceeds 50.000 active hosts.
To enable this capability you will need to install visual graphic stack. It will install and configure all daemons for you.
To enable metrics export to Clickhouse you need to run this command:
sudo fcli set main clickhouse_metrics true sudo fcli commit
Then you need to set database host:
sudo fcli set main clickhouse_metrics_host 127.0.0.1 sudo fcli commit
By default, we use Clickhouse on same machine but for large installations you may consider running it on another machine and in that case you will need to set external IP address. This field supports hostnames starting from 2.0.348. You can use both IPv4 and IPv6 addresses for clickhouse_metrics_host.
And then set port, we use native TCP/IP port for sending data to Clickhouse:
sudo fcli set main clickhouse_metrics_port 9000 sudo fcli commit
Set database name:
sudo fcli set main clickhouse_metrics_database fastnetmon sudo fcli commit
Optionally, you may set username and password:
sudo fcli set main clickhouse_metrics_username default sudo fcli set main clickhouse_metrics_password "" sudo fcli commit
You may even configure how often FastNetMon export traffic to Clickhouse, by default it does it every single second to provide real time graphs:
sudo fcli set main clickhouse_metrics_push_period 1 sudo fcli commit
FastNetMon does not rely on pre-created tables and it can create all required tables in Clickhouse automatically for you.
Clickhouse will create following tables for metrics export:
- total_metrics, total_metrics_ipv6
- network_metrics, network_metrics_ipv6
- host_metrics, host_metrics_ipv6
- asn_metrics_ipv4, asn_metrics_ipv6
To monitor that Clickhouse export works as expected you can check these counters:
sudo fcli show system_counters|grep click clickhouse_metrics_writes_total 2044855 clickhouse_metrics_writes_failed 0
Clickhouse may use significant amount of disk space and you can find detailed guide about ways to control disk space usage by it.
We offer following dashboards which will be installed by our instiller tool automatically.
Also, you could use such query in clickhouse-client tool to retrieve data:
select host, packets_incoming from fastnetmon.host_metrics where metricDate == today() AND metricDateTime >= now() - 5 order by packets_incoming desc limit 10;
If you use existing installation of Clickhouse you need to grant following permissions for Clickhouse user:
SELECT,INSERT,ALTER,CREATE TABLE, CREATE DICTIONARY
Top talkers pre-calculation
To provide lightning fast reports for top talker hosts in network FastNetMon has ability to pre-calculate hosts which generate most traffic each second and export traffic for them to Clickhouse.
This ability is enabled by default on all new installations. On old installations you can enable it this way:
sudo fcli set main clickhouse_metrics_export_top_hosts true sudo fcli commit
After enabling this option FastNetMon will create tables:
- host_metrics_ipv4_top
- host_metrics_ipv6_top
These tables are very similar to tables used to storing regular per host traffic: host_metrics, host_metrics_ipv6.
In these tables FastNetMon will store top 10 hosts in each particular second. It will use 4 different approaches to calculate top talkers:
- Top hosts by incoming bytes
- Top hosts by incoming packets
- Top hosts by outgoing bytes
- Top hosts by outgoing packets
If some hosts was picked up as top by more then one approach then we will store it only once.
Clickhouse password reset
In some rare cases you may need to reset password for Clickhouse user used to access database from Grafana:
wget https://install.fastnetmon.com/installer -Oinstaller sudo chmod +x installer sudo ./installer -reset_clickhouse_password
Due to Grafana limitations about inability to reload datasource configuration change you may need to run following commands to reload configuration:
sudo apt install -y sqlite3 sudo sqlite3 /var/lib/grafana/grafana.db
And then you will nee to remove cached datasource:
delete from data_source where uid = 'fastnetmonclickhouseofficialdatasourceuid';
And after that restart Grafana:
sudo systemctl restart grafana-server
After all these changes you will be able to access Clickhouse dashboards in Grafana.