FastNetMon Advanced relies on MongoDB for configuration storage. It does not store traffic here or metrics but it stores every configuration option. MongoDB is crucial for FastNetMon to function properly.
FastNetMon uses upstream version of MongoDB from official site.
By default both FastNetMon daemon and fcli command line tool connect directly to MongoDB. Both tools use login “fastnetmon_user” and password from file /etc/fastnetmon/keychain/.mongo_fastnetmon_password.
For debugging purposes you can login into MongoDB with admin password following way:
mongo admin --username administrator --password `sudo cat /etc/fastnetmon/keychain/.mongo_admin` use fastnetmon
For new installations you will need to use mongosh tool for management:
mongosh --username administrator --password `sudo cat /etc/fastnetmon/keychain/.mongo_admin` use fastnetmon
You can login with same permissions as fcli and FastNetMon use that way:
mongosh --username fastnetmon_user --password `sudo cat /etc/fastnetmon/keychain/.mongo_fastnetmon_password`
If you use FerretDB you may need to add following flag:
--authenticationMechanism PLAIN
If you use Docker based setup then you can get password that way:
sudo cat /var/lib/fastnetmon_docker/fastnetmon/conf/keychain/.mongo_admin
Then you will need to login into container with MongoDB:
sudo docker exec -ti mongo-server.networkd bash
And then use commands provided above manually setting password into arguments.
Optionally, you can set your own connection information for MongoDB using configuration file /etc/fastnetmon/fastnetmon.conf with following information:
{
"mongodb_host": "127.0.0.1",
"mongodb_port": 27017,
"mongodb_database_name": "fastnetmon",
"mongodb_username": "fastnetmon_user",
"mongodb_auth_source": "admin",
"mongodb_auth_mechanism": "",
"mongodb_timeout": 5,
"mongodb_timeout_heavy": 95
}
We use different timeout mongodb_timeout_heavy for queries which involve retrieval and processing of large amount of data. For all other queries we use mongodb_timeout.
After that, please put password for mongodb_username to file /etc/fastnetmon/keychain/.mongo_fastnetmon_password. Starting from version 2.0.351 you can use IPv6 address in field mongodb_host.
In addition to configuration storage FastNetMon may store attack information in JSON format. You can enable this behavior following way:
sudo fcli set main mongo_store_attack_information true sudo fcli commit
Attack information will be stored in collections with names attacks and mitigations. Data stored into attacks collection uses same format as web hook callbacks and JSON callback scripts.
For flow spec bans we store data into “mitigations” collection in following format:
{ "_id" : ObjectId("61254d9e09f2c264c71a152f"), "active" : true, "mitigation_source" : "automatic", "affected_host" : "0.0.0.0/32", "mitigation_type" : "flow_spec", "mitigation_uuid" : "f2411460-3904-4610-83f3-62632a9ba2a3", "details" : { "uuid" : "f2411460-3904-4610-83f3-62632a9ba2a3", "source_prefix" : "152.228.221.228/32", "destination_prefix" : "192.168.1.112/32", "destination_ports" : [ 34772 ], "source_ports" : [ 443 ], "packet_lengths" : [ 1492 ], "protocols" : [ "tcp" ], "fragmentation_flags" : [ "dont-fragment" ], "tcp_flags" : [ "ack|push" ], "action_type" : "discard", "action" : { } } }
For blackhole bans we store it in “attacks” collection using following format for ban:
{
_id: ObjectId('66fea5ad7695a083eb0450a8'),
action: 'ban',
alert_scope: 'host',
attack_details: {
attack_detection_source: 'manual',
attack_detection_threshold: 'unknown',
attack_detection_threshold_direction: 'unknown',
attack_direction: 'other',
attack_protocol: 'hopopt',
attack_severity: 'middle',
attack_type: 'unknown',
attack_uuid: '6642f712-e3c4-478f-8b3d-e400d176c12d',
average_incoming_flows: 0,
average_incoming_pps: 0,
average_incoming_traffic: 0,
average_incoming_traffic_bits: 0,
average_outgoing_flows: 0,
average_outgoing_pps: 0,
average_outgoing_traffic: 0,
average_outgoing_traffic_bits: 0,
host_group: 'global',
host_network: '45.189.200.0/22',
incoming_dropped_pps: 0,
incoming_dropped_traffic: 0,
incoming_dropped_traffic_bits: 0,
incoming_icmp_pps: 0,
incoming_icmp_traffic: 0,
incoming_icmp_traffic_bits: 0,
incoming_ip_fragmented_pps: 0,
incoming_ip_fragmented_traffic: 0,
incoming_ip_fragmented_traffic_bits: 0,
incoming_syn_tcp_pps: 0,
incoming_syn_tcp_traffic: 0,
incoming_syn_tcp_traffic_bits: 0,
incoming_tcp_pps: 0,
incoming_tcp_traffic: 0,
incoming_tcp_traffic_bits: 0,
incoming_udp_pps: 0,
incoming_udp_traffic: 0,
incoming_udp_traffic_bits: 0,
initial_attack_power: 0,
outgoing_dropped_pps: 0,
outgoing_dropped_traffic: 0,
outgoing_dropped_traffic_bits: 0,
outgoing_icmp_pps: 0,
outgoing_icmp_traffic: 0,
outgoing_icmp_traffic_bits: 0,
outgoing_ip_fragmented_pps: 0,
outgoing_ip_fragmented_traffic: 0,
outgoing_ip_fragmented_traffic_bits: 0,
outgoing_syn_tcp_pps: 0,
outgoing_syn_tcp_traffic: 0,
outgoing_syn_tcp_traffic_bits: 0,
outgoing_tcp_pps: 0,
outgoing_tcp_traffic: 0,
outgoing_tcp_traffic_bits: 0,
outgoing_udp_pps: 0,
outgoing_udp_traffic: 0,
outgoing_udp_traffic_bits: 0,
parent_host_group: '',
peak_attack_power: 0,
protocol_version: 'IPv4',
total_incoming_flows: 0,
total_incoming_pps: 0,
total_incoming_traffic: 0,
total_incoming_traffic_bits: 0,
total_outgoing_flows: 0,
total_outgoing_pps: 0,
total_outgoing_traffic: 0,
total_outgoing_traffic_bits: 0
},
ip: '1.2.3.4'
}
For blackhole unbans we store it this way (starting from v2.0.367):
{
_id: ObjectId('66fead2af5cb9c63ba0d6a3b'),
action: 'unban',
alert_scope: 'host',
attack_details: {
attack_detection_source: 'manual',
attack_detection_threshold: 'unknown',
attack_detection_threshold_direction: 'unknown',
attack_direction: 'other',
attack_protocol: 'unknown',
attack_severity: 'middle',
attack_type: 'unknown',
attack_uuid: '4d07be25-e458-4355-952f-aad22b7bd8d7',
average_incoming_flows: 0,
average_incoming_pps: 0,
average_incoming_traffic: 0,
average_incoming_traffic_bits: 0,
average_outgoing_flows: 0,
average_outgoing_pps: 0,
average_outgoing_traffic: 0,
average_outgoing_traffic_bits: 0,
host_group: 'global',
host_network: '45.189.200.0/22',
incoming_dropped_pps: 0,
incoming_dropped_traffic: 0,
incoming_dropped_traffic_bits: 0,
incoming_icmp_pps: 0,
incoming_icmp_traffic: 0,
incoming_icmp_traffic_bits: 0,
incoming_ip_fragmented_pps: 0,
incoming_ip_fragmented_traffic: 0,
incoming_ip_fragmented_traffic_bits: 0,
incoming_syn_tcp_pps: 0,
incoming_syn_tcp_traffic: 0,
incoming_syn_tcp_traffic_bits: 0,
incoming_tcp_pps: 0,
incoming_tcp_traffic: 0,
incoming_tcp_traffic_bits: 0,
incoming_udp_pps: 0,
incoming_udp_traffic: 0,
incoming_udp_traffic_bits: 0,
initial_attack_power: 0,
outgoing_dropped_pps: 0,
outgoing_dropped_traffic: 0,
outgoing_dropped_traffic_bits: 0,
outgoing_icmp_pps: 0,
outgoing_icmp_traffic: 0,
outgoing_icmp_traffic_bits: 0,
outgoing_ip_fragmented_pps: 0,
outgoing_ip_fragmented_traffic: 0,
outgoing_ip_fragmented_traffic_bits: 0,
outgoing_syn_tcp_pps: 0,
outgoing_syn_tcp_traffic: 0,
outgoing_syn_tcp_traffic_bits: 0,
outgoing_tcp_pps: 0,
outgoing_tcp_traffic: 0,
outgoing_tcp_traffic_bits: 0,
outgoing_udp_pps: 0,
outgoing_udp_traffic: 0,
outgoing_udp_traffic_bits: 0,
parent_host_group: '',
peak_attack_power: 0,
protocol_version: 'IPv4',
total_incoming_flows: 0,
total_incoming_pps: 0,
total_incoming_traffic: 0,
total_incoming_traffic_bits: 0,
total_outgoing_flows: 0,
total_outgoing_pps: 0,
total_outgoing_traffic: 0,
total_outgoing_traffic_bits: 0
},
ip: '2.3.4.5'
}
FastNetMon does not store information about time when it discovered attack and added it into MongoDB. But you can use internal feature of MongoDB to retrieve this information. For example, you have this attack in MongoDB (attacks collection):
db.attacks.findOne()
{
"_id" : ObjectId("5a985f3cec7006077668a321"),
"ip" : ":0001",
"attack_details" : {
"attack_uuid" : "e9ba83bd-389f-4332-8e2e-0300c01c839b",
"attack_severity" : "middle",
"attack_type" : "unknown",
....
}
}
And you can extract information when this object was added this way:
ObjectId("5a985f3cec7006077668a321").getTimestamp()
And it will return timestamp object:
ISODate("2022-03-01T20:14:52Z")
FastNetMon has logic to try connecting to MongoDB multiple times and you may find it in log file /var/log/fastnetmon/fastnetmon.log:
2022-08-11 12:28:20,459 [INFO] This is 1 try to connect to MongoDB 2022-08-11 12:28:20,459 [DEBUG] Start MongoDB connection checking 2022-08-11 12:28:20,459 [INFO] PING mongodb database fastnetmon 2022-08-11 12:28:20,460 [ERROR] Could not ping MongoDB: No suitable servers found (`serverSelectionTryOnce` set): [connection refused calling ismaster on '127.0.0.1:27017']: generic server error 2022-08-11 12:28:20,460 [INFO] Will do second try in 5 seconds 2022-08-11 12:28:25,460 [INFO] This is 2 try to connect to MongoDB 2022-08-11 12:28:25,460 [DEBUG] Start MongoDB connection checking 2022-08-11 12:28:25,460 [INFO] PING mongodb database fastnetmon 2022-08-11 12:28:25,460 [ERROR] Could not ping MongoDB: No suitable servers found (`serverSelectionTryOnce` set): [connection refused calling ismaster on '127.0.0.1:27017']: generic server error 2022-08-11 12:28:25,460 [INFO] Will do second try in 5 seconds 2022-08-11 12:28:30,460 [INFO] This is 3 try to connect to MongoDB 2022-08-11 12:28:30,460 [DEBUG] Start MongoDB connection checking 2022-08-11 12:28:30,460 [INFO] PING mongodb database fastnetmon 2022-08-11 12:28:30,461 [ERROR] Could not ping MongoDB: No suitable servers found (`serverSelectionTryOnce` set): [connection refused calling ismaster on '127.0.0.1:27017']: generic server error 2022-08-11 12:28:30,461 [INFO] Will do second try in 5 seconds
fcli has similar logic and it tries multiple times to establish connection to MongoDB:
sudo fcli
2022/08/11 12:48:35 We cannot establish connection with MongoDB: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: 127.0.0.1:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: dial tcp 127.0.0.1:27017: connect: connection refused }, ] } Attempt number: 1
To investigate root cause of issues with MongoDB we recommend checking daemon status first:
sudo systemctl status mongod
And output for MongoDB instance looks like:
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-08-11 12:29:46 BST; 3s ago
Docs: https://docs.mongodb.org/manual
Main PID: 8921 (mongod)
CGroup: /system.slice/mongod.service
└─8921 /usr/bin/mongod --config /etc/mongod.conf
Aug 11 12:29:46 fastlab1 systemd[1]: Started MongoDB Database Server.
If “Active” field differs from “active” then you may have issues with MongoDB.
In addition to systemd service status you may check that MongoDB process is running:
ps aux|grep mongo mongodb 8921 0.3 0.2 1009456 73424 ? Ssl 12:29 0:03 /usr/bin/mongod --config /etc/mongod.conf odintsov 9465 0.0 0.0 12940 968 pts/0 S+ 12:47 0:00 grep --color=auto mongo
First thing to check is your disk space usage. You need to have at least 10G of spare disk space for MongoDB to operate correctly, you can check disk space usage this way:
df -h
To find more reasons about issues with MongoDB we recommend checking log file /var/log/mongodb/mongod.log.
In addition to log file you may check systemd service logs this way:
sudo journalctl -u mongod -n 1000 -f
In case when your server runs out of disk space MongoDB terminates it’s daemons to prevent data corruption and you will need to start it again manually. You need to allocate more disk space to server or free up existing disk space. We recommend carefully investigating which service used all disk space to avoid same cases in future. It can be done using following command:
sudo systemctl start mongod
And then you need to restart FastNetMon as it depends on MongoDB:
sudo systemctl restart fastnetmon
You can check that FastNetMon is running well using fcli command:
sudo fcli show total_traffic_counters
Modern versions of MongoDB require AVX support and if CPU on your server does not support it then it will crash with SIGILL error like this:
sudo service mongod status × mongod.service - MongoDB Database Server Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled) Active: failed (Result: signal) since Thu 2023-11-09 14:56:28 UTC; 5min ago Docs: https://docs.mongodb.org/manual Process: 154878 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=killed, signal=ILL) Main PID: 154878 (code=killed, signal=ILL) CPU: 12ms Nov 09 14:56:28 flowdetector systemd[1]: Started MongoDB Database Server. Nov 09 14:56:28 flowdetector systemd[1]: mongod.service: Main process exited, code=killed, status=4/ILL Nov 09 14:56:28 flowdetector systemd[1]: mongod.service: Failed with result 'signal'.
Upgrade procedure
If you run old version of MongoDB and you want to upgrade to latest one you cannot do upgrade directly to latest version due to MongoDB upgrade process constraints. You need to do upgrade version to version. Fortunately, you can skip some versions where MongoDB database format did not change.
Before you start upgrade process we recommend doing full backup of VM / server and making configuration export this way.
Let’s assume that you run MongoDB 3.4. To upgrade to 5.0 which is relatively new and supported version you need to do following upgrades:
- Upgrade 3.4 to 3.6: documentation
- Upgrade 3.6 to 4.0: documentation
- Upgrade 4.0 to 4.2: documentation
- Upgrade 4.2 to 4.4: documentation
- Upgrade 4.4 to 5.0: documentation
Removal of large number of values from configuration
If you’ve added too many networks in configuration and looking for easy way to remove all of them you can use following command in MongoDB client:
db.configuration.findOneAndUpdate({}, { $set: { networks_list: [] } });
How could I backup whole MongoDB database?
We recommend using our own backup tool.
export FASTNETMON_BACKUP_PATH=/var/backup/fastnetmon-`date +"%m-%d-%Y-%T"` sudo mkdir $FASTNETMON_BACKUP_PATH sudo mongodump --username administrator --password `sudo cat /etc/fastnetmon/keychain/.mongo_admin` --out $FASTNETMON_BACKUP_PATH tar -cpzf ~/fastnetmon_configuration_dump.tar.gz $FASTNETMON_BACKUP_PATH
How could I import MongoDB backup?
We recommend using our own backup tool.
Be careful. This command will overwrite current configuration with data from backup!
sudo mongorestore --username administrator --password `sudo cat /etc/fastnetmon/keychain/.mongo_admin` --drop /tmp/folder_with_incompressed_backup
And ask FastNetMon to re-read new configuration:
sudo fcli commit
Controlling MongoDB verbosity in logs
If you find that MongoDB generates excessive amount of log messages to /var/log/mongodb/mongod.log you can decrease verbosity level by doing following change (adding quiet flag) in /etc/mongod.conf.
systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log # Reduce logging verbosity little bit quiet: true
For all new installations this is default configuration.
Unfortunately, this change does not stop even latest MongoDB 6.0.13 from logging every single client connection to log file this way:
{"t":{"$date":"2024-03-06T15:39:51.328+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn11","msg":"Interrupted operation as its client disconnected","attr":{"opId":23223}}
{"t":{"$date":"2024-03-06T15:39:51.772+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn14","msg":"client metadata","attr":{"remote":"127.0.0.1:55784","client":"conn14","doc":{"driver":{"name":"mongo-go-driver","version":"v1.12.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.18.1"}}}
{"t":{"$date":"2024-03-06T15:39:51.772+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn13","msg":"client metadata","attr":{"remote":"127.0.0.1:55782","client":"conn13","doc":{"driver":{"name":"mongo-go-driver","version":"v1.12.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.18.1"}}}
{"t":{"$date":"2024-03-06T15:39:51.773+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn15","msg":"client metadata","attr":{"remote":"127.0.0.1:55796","client":"conn15","doc":{"driver":{"name":"mongo-go-driver","version":"v1.12.1"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.18.1"}}}
MongoDB logrotate configuration
Please note that this change is included in FastNetMon 2.0.362 and newer releases.
To address issue with MongoDB excessive logging we can recommend enabling logrotate for MongoDB log file /var/log/mongodb/mongod.log.
As first step please make following changes to MongoDB configuration file /etc/mongod.conf:
systemLog: destination: file logAppend: true logRotate: reopen path: /var/log/mongodb/mongod.log # Reduce logging verbosity little bit quiet: true
Then apply configuration changes:
sudo systemctl restart mongod
Then create following file /etc/logrotate.d/mongodb with following content:
/var/log/mongodb/mongod.log
{
rotate 7
daily
size 1M
missingok
create 0600 mongodb mongodb
delaycompress
compress
sharedscripts
postrotate
/bin/kill -SIGUSR1 $(pidof mongod)
endscript
}
To check that it works you can run following actions:
sudo logrotate --force /etc/logrotate.d/mongodb