09.12.2017

FastNetMon cookbook

Introduction

In this document, you could find number of different useful tips about FastNetMon Advanced.

Configure FastNetMon to use external InfluxDB instance

set main influxdb_database fastnetmon 
set main influxdb_host 127.0.0.1 
set main influxdb_port 8086 
set main influxdb_push_period 1 
set main graphite disable 
commit

Upgrade FastNetMon to next stable release

sudo apt-get update
sudo apt-get install --only-upgrade fastnetmon

And then you should apply configuration changes:

sudo fcli upgrade_configuration

How can I export or import FastNetMon’s configuration from backup?

Please follow this guide

How to login into MongoDB?

mongo admin --username administrator --password `sudo cat /etc/fastnetmon/keychain/.mongo_admin`
use fastnetmon

How could I backup configuration?

export FASTNETMON_BACKUP_PATH=/var/backup/fastnetmon-`date +"%m-%d-%Y-%T"`
sudo mkdir $FASTNETMON_BACKUP_PATH
sudo mongodump  --username administrator --password `sudo cat /etc/fastnetmon/keychain/.mongo_admin` --out $FASTNETMON_BACKUP_PATH
tar -cpzf ~/fastnetmon_configuration_dump.tar.gz $FASTNETMON_BACKUP_PATH

How could I import backup?

Be careful! This command will overwrite current configuration with data from backup!

sudo mongorestore --username administrator --password `sudo cat /etc/fastnetmon/keychain/.mongo_admin`  --drop /tmp/folder_with_incompressed_backup

And ask FastNetMon to re-read new configuration:

sudo fcli commit

How could I use custom host/port in fcli?

If you use not standard port and host for API you could specify them using environment variables:

API_PORT=46551 API_HOST=127.0.0.7 sudo -E -i fcli

Do you have replacement tool for fastnetmon_client?

Yes, we offer improved version of fastnetmon_client with IPv6 support.

I want to use custom connection details for Mongo DB.

You could create special file for fcli and fastnetmon advanced core to specify custom connection details:

sudo vim /etc/fastnetmon/fastnetmon.conf

And put following:

{
  "mongodb_host": "127.0.0.1",
  "mongodb_port": 27017,
  "mongodb_database_name": "fastnetmon",
  "mongodb_username": "fastnetmon_user",
  "mongodb_auth_source": "admin"
}

After that, please put password for mongodb_username to file /etc/fastnetmon/keychain/.mongo_fastnetmon_password.

I have very big number of networks and do not want to specify them manually

You could use Radb data for this task. You could use whois tool and make following query for 2 byte (16 bit) ASN numbers:

whois -h whois.radb.net '!gasXXXX'

For 32 bit (4 byte ) ASN you could use tool to convert them into two 16 bit numbers delimited by dot: http://submit.apnic.net/cgi-bin/convert-asn.pl

Then you could use following syntax:

whois -h whois.radb.net '!gasXXXX.XXXX'

Then you could use aggregate tool for aggregation. It’s available in Ubuntu repositories:

sudo apt-get install -y aggregate

How to enable ASN lookups for traffic DB?

Run this tool and fill dictionaries for ASN (please be prepared to wait 5-10 minutes):

sudo fill_dictionaries

And then reload traffic_db daemon:

sudo service traffic_db restart

Then you could check log file /var/log/fastnetmon/traffic_db.log  for line: “I found that you have ASN database. I will use it”

Could I use nfcapd or other Netflow forwarders?

Yes, you could. But all your devices should have unique source_id. Because we use agent_ip + source_id as unique identifier for templates. If you use Netflow forwarders/aggregators they replace agent’s IP by forwarder’s IP and you could hit templates conflict. Result of this conflict could be very dangerous: completely incorrect traffic processing or tool’s failures in some rare cases.

How could I select top K hosts by traffic in InfluxDB?

use fastnetmon 
select top(packets_incoming, host, 10),host from hosts_traffic where time > now() - 1h;

How could I select top K network by traffic amount in InfluxDB?

use fastnetmon;
select top(packets_incoming, network, 10),network from networks_traffic where time > now() - 1h;

How to create host group?

To create new group:

sudo fcli set hostgroup my_new_group
sudo fcli set hostgroup my_new_group threshold_mbps 100
sudo fcli set hostgroup my_new_group ban_for_bandwidth enable

Enable ban actions for my_new_group host group:

sudo fcli set hostgroup my_new_group enable_ban enable

And then add required networks:

sudo fcli set hostgroup my_new_group networks 11.22.33.44/24

And finally commit changes:

sudo fcli commit

I could not find traffic data in InfluxDB for particular hosts

First of all, please check that FNM sees traffic for this host. If a host has zero traffic we will not push data into InfluxDB.

If you have traffic then please check InfluxDB logs /var/log/influxdb/influxd.log for record like “WARN: 100% of max-values-per-tag limit exceeded: (100096/100000), db=fastnetmon shard=7 measurement=hosts_traffic tag=host”.

This issue could be fixed by increasing max-values-per-tag value in /etc/influxdb/influxdb.conf and InfluxDB restart:

sudo service influxdb restart

How flow_spec_execute_validation works?

Validation includes following steps:

  • Source prefix should be exactly /32
  • Destination prefix should be exactly /32
  • Source prefix should belong to networks_list
  • Destination prefix should belong to networks_list

You could disable it this way:

sudo fcli set main flow_spec_execute_validation disable
sudo fcli commit

How I could remove all networks from networks_list?

for i in `sudo fcli show main networks_list`;do sudo fcli delete main networks_list $i;done

How could I remove all host groups?

for i in `sudo fcli show hostgroup|egrep "^name"|awk '{print $2}'`;do sudo fcli delete hostgroup $i;done

Selective BGP blackhole or traffic diversion

Please check this guide if you need /32 or /24 announces for different host groups.

sFlow, Netflow or IPFIX traffic duplication to multiple sources

We have detailed guide about it.

How can I find time when attack information was added into MongoDB?

FastNetMon does not have information about time when it discovered attack and added it into MongoDB.

But you can use internal feature of MongoDB to retrieve this information. For example, you have this attack in MongoDB (attacks collection):

db.attacks.findOne()
{
	"_id" : ObjectId("5a985f3cec7006077668a321"),
	"ip" : ":0001",
	"attack_details" : {
		"attack_uuid" : "e9ba83bd-389f-4332-8e2e-0300c01c839b",
		"attack_severity" : "middle",
		"attack_type" : "unknown",
                ....
	}
}

And you can extract information when this object was added this way:

ObjectId("5a985f3cec7006077668a321").getTimestamp()

And it will return timestamp object:

ISODate("2018-03-01T20:14:52Z")

Can I use session on top of IPv6 addresses?

Yes, you can. Just specify IPv6 addresses for local_address and remote_address configuration options. You can announce IPv4 and IPv6 announces over peering connection on top of IPv6 addresses and you can do same on top of peering connection on IPv4 addresses.

How can I configure baseline / threshold values according to historical data?

You can use this guide as reference. Also, we have significantly improved version based on Clickhouse.

InfluxDB consumes significant amount of disk space. Can I reduce disk requirements?

Yes, you can. If you do not need per host counters you can follow this guide. If you can’t do it, you can reduce amount of stored data this way.

Can I have multi-level thresholds?

Yes, you can implement them using API. We have complete guide.

Is it possible to change configuration for traffic_db?

Yes, it’s possible. We added number of options in 2.0.84 release, please upgrade before using these options.

You need to create file /etc/fastnetmon/traffic_db.conf:

{
"clickhouse_batch_size": 1000,
"clickhouse_batch_delay": 1,
"traffic_db_host":"127.0.0.1",
"traffic_db_port": 8100,
"clickhouse_host": "127.0.0.1",
"clickhouse_port": 9000,
"clickhouse_user": "default",
"clickhouse_password": ""
}

And apply changes:

sudo service traffic_db restart

Can FastNetMon export more details about bandwidth consumed by each traffic type?

Yes, FastNetMon can export more details to InfluxDB, please follow this guide to enable per protocol counters.

Do you have integration with Radware?

Yes, we have, please check this integration module and next version of integration (recommended for production):

Can we use HA mode for FastNetMon to handle failure of any FastNetMon instances?

Yes, you can, we offer number of options for it.

Can I read networks lists directly from BGP session?

Yes, you can. We have bundled feature for this task here

How well FastNetMon scales under real load?

We have detailed guide about it here

Adding big number of networks

If you have text file with all networks in CIDR format like this:

22.33.44.55/29
44.66.11.66/27

You can add all of them this way:

for i in `cat list`; do sudo fcli set hostgroup group_name networks $i;done

After executing this script, please commit changes:

sudo fcli commit

If you have duplicating networks, FastNetMon will ignore duplicates and report error this way:

Command returned error: This option is already exists

Feel free to ignore such warnings.

Do you have any deployment scenarios for FastNetMon Advanced

Yes, we have detailed guide about it.

Can you suggest best scenarios to deliver traffic information to FastNetMon over public networks?

Yes, we have detailed guide about it.

Do you have support for sampled Netflow and IPFIX?

That’s quite tricky area, please check our article about it.

Can you provide detailed algorithm used by FastNetMon for attack detection?

You can find detailed logic here

Can FastNetMon detect remote hosts which attack my network?

Yes, it can, please check this article.

Do you support AF_XDP?

Sure, we recently added support for it and you may try it using this guide.

Can I add another user for Grafana or reset password for existing users?

Generate reliable password using:

pwgen 16 1

And add user:

sudo htpasswd /etc/nginx/.htpasswd support

To reset password for default user, please use “admin” instead of “support”.

Can I keep blocked hosts during FastNetMon restarts?

Yes, you can. Please use this guide.

Can I run multiple notify scripts?

Yes, of course, you can. Please use this guide.

Can I install FastNetMon without using install tool?

Yes, you can do it using our guide.

I do not see all traffic in traffic persistence database. How can I debug it?

Under significant load traffic_db could drop packets due to small UDP buffers in default Linux configuration.

You can confirm this error using netstat tool:

sudo netstat -s -u

And check field “receive buffer errors” in output, it should increase every few seconds

To solve this issue we recommend following Linux configurations:

sudo vim /etc/sysctl.d/10-udp-buffers.conf

And add following options:

net.core.rmem_default = 2129920
net.core.rmem_max = 21299200

Apply changes:

sudo sysctl --system

After these changes please restart FastNetMon and traffic_db:

sudo service fastnetmon restart
sudo service traffic_db restart

Can I whitelist arbitrary traffic from FastNetMon?

Since version 2.0.131 FastNetMon can whitelist any traffic from processing using rich set of rules. You can check syntax here (these rules do not use section “actions”).

Please add whitelist rules into file /etc/fastnetmon/whitelist_rules.dat. Please add single rule per line.

For example:

{ "destination_ports": [ 443 ], "protocols": [ "udp" ] }
{ "source_ports": [ 443 ], "protocols": [ "udp" ] }

FastNetMon will track such packets using counter total_flowspec_whitelist_packets (sudo fcli show system_counters).

This traffic will be completely discarded from processing and completely ignored. Be very careful with this option because it may cause issues with DDoS detection.

Can I use InfluxDB with authentication?

Yes, from version 2.0.132 you can enable authentication in InfluxDB this way:

sudo fcli set main influxdb_auth enable
sudo fcli set main influxdb_user admin
sudo fcli set main influxdb_password influxsecure999

From InfluxDB side you have to enable auth in /etc/influxdb/influxdb.conf

auth-enabled = true

Then restart daemon:

service influxdb restart

And create admin user:

influx
CREATE USER admin WITH PASSWORD 'influxsecure999' WITH ALL PRIVILEGES

Can I convert configuration from community edition to advanced?

Yes, you can, please use this guide.

How can I debug CPU issues with FastNetMon without using external monitoring tools?

You can use Telegraf and our dashboard from this guide: guide.

Can FastNetMon inject BGP announces received from external data source?

Yes, it can, please check this article.