FastNetMon Advanced Traffic Persistence

Introduction

FastNetMon has bundled ability to store all traffic inside a special column-oriented database. You could store all traffic inside it and retrieve it using a well-known SQL syntax.

System requirements

You should have a working instance of FastNetMon before using this guide. Also, you need to install visual traffic stack because we use Grafana for traffic persistence.

Configure FastNetMon to store traffic in the traffic database

sudo fcli set main traffic_db_host 127.0.0.1
sudo fcli set main traffic_db_port 8100
sudo fcli set main traffic_db enable
sudo fcli commit

If you run port mirror mode, you may need to adjust this value, which has the purpose of reducing load on the database and sampling all incoming data in addition to the sampling rate configured on port mirror plugin side. If you run port mirror with a reasonable sampling rate, then you can set this value to 1.

sudo fcli set main traffic_db_sampling_rate 1
sudo fcli commit

And start traffic_db daemon:

sudo systemctl restart traffic_db

Traffic database daemon uses its own configuration file /etc/fastnetmon/traffic_db.conf, and you can change default options as follows:

{
"traffic_db_host":"127.0.0.1",
"traffic_db_port": 8100,
"clickhouse_batch_size": 1000,
"clickhouse_batch_delay": 1,
"clickhouse_host": "127.0.0.1",
"clickhouse_port": 9000,
"clickhouse_user": "default",
"clickhouse_password": "",
"clickhouse_database_name": "fastnetmon",
"clickhouse_table_name": "traffic"
}

Please note that options clickhouse_database_name and clickhouse_table_name were added in 2.0.363.

After making any changes, you need to restart the daemon.

sudo systemctl restart traffic_db

Making example queries

Start client:

clickhouse-client --host 127.0.0.1

Example query:

select * from fastnetmon.traffic limit 10;

Execute ASN query:

use fastnetmon
SELECT dstAsn, count(dstAsn) AS cnt FROM traffic GROUP BY dstAsn  ORDER BY cnt DESC  LIMIT 10

As output, you could find the number of packets per destination ASN for required time slice.

Clickhouse has logic (TTL) in place to remove data which is older than 7 days.

You can find the SQL schema used for the traffic persistence database called traffic:

CREATE TABLE fastnetmon.traffic
(
    `packetDate` Date DEFAULT toDate(packetDateTime),
    `packetDateTime` DateTime,
    `source` Int8,
    `sampleRatio` UInt32,
    `srcIp` UInt32,
    `dstIp` UInt32,
    `srcIpv6` FixedString(16),
    `dstIpv6` FixedString(16),
    `srcAsn` UInt32,
    `dstAsn` UInt32,
    `inputInterface` UInt32,
    `outputInterface` UInt32,
    `ipProtocolVersion` UInt8,
    `ttl` UInt8,
    `sourcePort` UInt16,
    `destinationPort` UInt16,
    `protocol` UInt32,
    `length` UInt64,
    `numberOfPackets` UInt64,
    `flags` UInt8,
    `ipFragmented` Int8,
    `ipDontFragment` Int8,
    `packetPayloadLength` Int32,
    `packetPayloadFullLength` UInt32,
    `packetDirection` Int8,
    `agentIpAddress` UInt32,
    `srcMac` FixedString(6),
    `dstMac` FixedString(6)
)
ENGINE = MergeTree
PARTITION BY packetDate
ORDER BY (srcIp, dstIp, packetDate)
TTL packetDate + toIntervalDay(7)
SETTINGS index_granularity = 8192

PacketDirection uses the following integers to encode directions:

• Incoming 0
• Outgoing 1
• Internal 2
• Other 3

Source uses the following integers to encode the plugin which captured traffic:

• Unknown 0
• Mirror 1
• sFlow = 2
• Netflow / IPFIX = 3
• FastNetMon Flow = 4

Default schema uses index configuration, which may not provide the best performance for all types of queries. You can alter the index configuration and then re-create the table using preferred index granularity and configuration for your case. You just need to keep all fields as-is.

After making changes in schema you will need to restart the traffic_db daemon this way:

sudo systemctl restart traffic_db

As another option to speed up certain queries, you can copy all data from the original traffic table to another table with a different index configuration using the MATERIALIZED VIEW capability:

CREATE MATERIALIZED VIEW fastnetmon.traffic_sampled ENGINE = MergeTree PARTITION BY packetDate ORDER BY (srcIp, dstIp, packetDate) TTL packetDate + toIntervalDay(7) SETTINGS index_granularity = 8192
populate as select * from fastnetmon.traffic WHERE rand() % 100 = 1;

In this example, we used sampling with rand() section to make queries even faster, but you can copy all the data without using sampling.

Example dashboards

Dashboards to access traffic flow information are installed by default when you install visual traffic stack.

If you’re interested in peering per ASN traffic reports, we recommend using this dashboard.

In case of any issues with traffic_db we recommend checking the following log file: /var/log/fastnetmon/traffic_db.log