FastNetMon Advanced Traffic Persistence

Introduction

FastNetMon has bundled ability to store all traffic inside special column-oriented database. You could store all traffic inside it and retrieve it using well-known SQL syntax.

System requirements

You should have working instance of FastNetMon before using this guide. Also, you need to install visual traffic stack because we use Grafana for traffic persistence too.

Configure FastNetMon to store traffic into traffic database

sudo fcli set main traffic_db_host 127.0.0.1
sudo fcli set main traffic_db_port 8100
sudo fcli set main traffic_db enable
sudo fcli commit

If you run port mirror mode then you may need to adjust this value which has a purpose to reduce load on database and sample all incoming data in addition to sampling rate configured on port mirror plugin side. If you run port mirror with reasonable sampling rate then you can set this value to 1.

sudo fcli set main traffic_db_sampling_rate 1
sudo fcli commit

And start traffic_db daemon:

sudo systemctl restart traffic_db

Traffic database daemon uses it’s own configuration file /etc/fastnetmon/traffic_db.conf and you may change default options here:

{
"traffic_db_host":"127.0.0.1",
"traffic_db_port": 8100,
"clickhouse_batch_size": 1000,
"clickhouse_batch_delay": 1,
"clickhouse_host": "127.0.0.1",
"clickhouse_port": 9000,
"clickhouse_user": "default",
"clickhouse_password": "",
"clickhouse_database_name": "fastnetmon",
"clickhouse_table_name": "traffic"
}

Please note that options clickhouse_database_name and clickhouse_table_name were added in 2.0.363.

After making any changes you need to restart daemon.

sudo systemctl restart traffic_db

Make example queries

Start client:

clickhouse-client --host 127.0.0.1

Example query:

select * from fastnetmon.traffic limit 10;

Execute ASN query:

use fastnetmon
SELECT dstAsn, count(dstAsn) AS cnt FROM traffic GROUP BY dstAsn  ORDER BY cnt DESC  LIMIT 10

As output, you could find number of packets per destination ASN for required time slice.

Clickhouse has logic (TTL) in place to remove data which is older then 7 days.

You can find SQL schema used for traffic persistence database called traffic:

CREATE TABLE fastnetmon.traffic
(
    `packetDate` Date DEFAULT toDate(packetDateTime),
    `packetDateTime` DateTime,
    `source` Int8,
    `sampleRatio` UInt32,
    `srcIp` UInt32,
    `dstIp` UInt32,
    `srcIpv6` FixedString(16),
    `dstIpv6` FixedString(16),
    `srcAsn` UInt32,
    `dstAsn` UInt32,
    `inputInterface` UInt32,
    `outputInterface` UInt32,
    `ipProtocolVersion` UInt8,
    `ttl` UInt8,
    `sourcePort` UInt16,
    `destinationPort` UInt16,
    `protocol` UInt32,
    `length` UInt64,
    `numberOfPackets` UInt64,
    `flags` UInt8,
    `ipFragmented` Int8,
    `ipDontFragment` Int8,
    `packetPayloadLength` Int32,
    `packetPayloadFullLength` UInt32,
    `packetDirection` Int8,
    `agentIpAddress` UInt32,
    `srcMac` FixedString(6),
    `dstMac` FixedString(6)
)
ENGINE = MergeTree
PARTITION BY packetDate
ORDER BY (srcIp, dstIp, packetDate)
TTL packetDate + toIntervalDay(7)
SETTINGS index_granularity = 8192

PacketDirection uses following integers to encode directions:

• Incoming 0
• Outgoing 1
• Internal 2
• Other 3

Source uses following integers to encode plugin which captured traffic:

• Unknown 0
• Mirror 1
• sFlow = 2
• Netflow / IPFIX = 3
• FastNetMon Flow = 4

Default schema uses index configuration which may not provide best performance for all types queries. You can alter index configuration and then re-create table using preferred index granularity and configuration for your case. You just need to keep all fields as-is.

After making changes in schema you will need to restart traffic_db daemon this way:

sudo systemctl restart traffic_db

As another option to speed up some queries you can copy all data from original traffic table to another table with different index configuration using MATERIALIZED VIEW capability:

CREATE MATERIALIZED VIEW fastnetmon.traffic_sampled ENGINE = MergeTree PARTITION BY packetDate ORDER BY (srcIp, dstIp, packetDate) TTL packetDate + toIntervalDay(7) SETTINGS index_granularity = 8192
populate as select * from fastnetmon.traffic WHERE rand() % 100 = 1;

In this example we used sampling with rand() section to make queries even faster but you can copy all data without using sampling.

Example dashboards

Dashboards to access traffic flow information are installed by default when you install visual traffic stack.

If you’re interested in peering per ASN traffic reports we recommend using this new dashboard.

In case of any issues with traffic_db we recommend checking following log file: /var/log/fastnetmon/traffic_db.log