Please note that this capability is a part of partner only integration feature set and we strongly advice against using it unless you have discussed it with our engineering team.
This logic can be changed or removed without any further notices and it’s not a part of FastNetMon backward compatibility guarantee.
To enable per hostgroup traffic profile collection logic you will need to enable these configuration options:
sudo fcli set main generate_hostgroup_traffic_samples true sudo fcli set main generate_hostgroup_traffic_samples_delay 60 sudo fcli commit
After that FastNetMon will calculate amount of traffic with drop down by type of traffic for each per_host hostgroup you have in configuration and will store them to MongoDB in collection: hostgroup_traffic_stats. It will do it every generate_hostgroup_traffic_samples_delay seconds.
FastNetMon exports amount of transferred traffic using following fields:
- source_ip
- destination_ip
- protocol
- ttl
- packet_length
- tcp_flags
- source_port
- destination_port
- source_country
- destination_country
- source_asn
- destination_asn
- source_interface
- destination_interface
- device_address
Please note that this capability depends on correctly configured traffic buffer and if you forget to configure it you will see following error in /var/log/fastnetmon/fastnetmon.log
Please enable traffic buffer to generate hostgroup traffic profiles
To get them in MongoDB you can use following query:
db.hostgroup_traffic_stats.find({})
Example report looks this way:
[
{
_id: ObjectId('66d9d598dc1c421e770e3426'),
fields: [
{
bytes: 5760183,
ordered_by: 'packets',
packets: 39064,
source_ip: '1.2.3.4'
},
....
{
bytes: 44638204,
elements: 1984,
packets: 118284,
source_ip: 'total'
},
...
{
bytes: 15005,
destination_ip: '2.3.4.5',
ordered_by: 'bytes',
packets: 44
},
{
bytes: 44638204,
destination_ip: 'total',
elements: 1449,
packets: 118284
},
{
bytes: 25518912,
ordered_by: 'packets',
packets: 87422,
protocol: '6'
},
{
bytes: 18890272,
ordered_by: 'packets',
packets: 27083,
protocol: '17'
},
{
bytes: 218682,
ordered_by: 'packets',
packets: 3712,
protocol: '1'
},
{
bytes: 4600,
ordered_by: 'packets',
packets: 37,
protocol: '50'
},
{
bytes: 5738,
ordered_by: 'packets',
packets: 30,
protocol: '47'
},
{
bytes: 25518912,
ordered_by: 'bytes',
packets: 87422,
protocol: '6'
},
{
bytes: 18890272,
ordered_by: 'bytes',
packets: 27083,
protocol: '17'
},
{
bytes: 218682,
ordered_by: 'bytes',
packets: 3712,
protocol: '1'
},
{ bytes: 5738, ordered_by: 'bytes', packets: 30, protocol: '47' },
{ bytes: 4600, ordered_by: 'bytes', packets: 37, protocol: '50' },
{
bytes: 44638204,
elements: 5,
packets: 118284,
protocol: 'total'
},
{
bytes: 44638204,
ordered_by: 'packets',
packets: 118284,
ttl: '0'
},
{
bytes: 44638204,
ordered_by: 'bytes',
packets: 118284,
ttl: '0'
},
{ bytes: 44638204, elements: 1, packets: 118284, ttl: 'total' },
{
bytes: 43249862,
ordered_by: 'packets',
packet_length: 'jumbo_packets',
packets: 103514
},
{
bytes: 732127,
ordered_by: 'packets',
packet_length: '0_150',
packets: 10097
},
{
bytes: 224152,
ordered_by: 'packets',
packet_length: '151_300',
packets: 1655
},
{
bytes: 115167,
ordered_by: 'packets',
packet_length: '301_450',
packets: 747
},
{
bytes: 75581,
ordered_by: 'packets',
packet_length: '751_900',
packets: 725
},
{
bytes: 50099,
ordered_by: 'packets',
packet_length: '1051_1200',
packets: 391
},
{
bytes: 47632,
ordered_by: 'packets',
packet_length: '601_750',
packets: 361
},
{
bytes: 23177,
ordered_by: 'packets',
packet_length: '1201_1350',
packets: 225
},
{
bytes: 70347,
ordered_by: 'packets',
packet_length: '451_600',
packets: 222
},
{
bytes: 26153,
ordered_by: 'packets',
packet_length: '901_1050',
packets: 212
},
{
bytes: 23907,
ordered_by: 'packets',
packet_length: '1351_1500',
packets: 135
},
{
bytes: 43249862,
ordered_by: 'bytes',
packet_length: 'jumbo_packets',
packets: 103514
},
{
bytes: 732127,
ordered_by: 'bytes',
packet_length: '0_150',
packets: 10097
},
{
bytes: 224152,
ordered_by: 'bytes',
packet_length: '151_300',
packets: 1655
},
{
bytes: 115167,
ordered_by: 'bytes',
packet_length: '301_450',
packets: 747
},
{
bytes: 75581,
ordered_by: 'bytes',
packet_length: '751_900',
packets: 725
},
{
bytes: 70347,
ordered_by: 'bytes',
packet_length: '451_600',
packets: 222
},
{
bytes: 50099,
ordered_by: 'bytes',
packet_length: '1051_1200',
packets: 391
},
{
bytes: 47632,
ordered_by: 'bytes',
packet_length: '601_750',
packets: 361
},
{
bytes: 26153,
ordered_by: 'bytes',
packet_length: '901_1050',
packets: 212
},
{
bytes: 23907,
ordered_by: 'bytes',
packet_length: '1351_1500',
packets: 135
},
{
bytes: 23177,
ordered_by: 'bytes',
packet_length: '1201_1350',
packets: 225
},
{
bytes: 44638204,
elements: 11,
packet_length: 'total',
packets: 118284
},
{
bytes: 9807668,
ordered_by: 'packets',
packets: 35255,
tcp_flags: 'psh,ack'
},
{
bytes: 10749258,
ordered_by: 'packets',
packets: 30799,
tcp_flags: 'ack'
},
{
bytes: 1486870,
ordered_by: 'packets',
packets: 14286,
tcp_flags: 'syn'
},
{
bytes: 3327593,
ordered_by: 'packets',
packets: 5086,
tcp_flags: 'syn,ack'
},
...
{
bytes: 25518912,
elements: 253,
packets: 87422,
tcp_flags: 'total'
},
{
bytes: 17114707,
ordered_by: 'packets',
packets: 34290,
source_port: '8291'
},
...
{
bytes: 44409184,
elements: 2703,
packets: 114505,
source_port: 'total'
},
{
bytes: 3937222,
destination_port: '8291',
ordered_by: 'packets',
packets: 31008
},
...
{
bytes: 44409184,
destination_port: 'total',
elements: 4317,
packets: 114505
},
{
bytes: 14869861,
ordered_by: 'packets',
packets: 49528,
source_asn: '269872'
},
{
bytes: 44638204,
elements: 252,
packets: 118284,
source_asn: 'total'
},
{
bytes: 29768343,
destination_asn: '269872',
ordered_by: 'packets',
packets: 68756
},
...
{
bytes: 44638204,
destination_asn: 'total',
elements: 174,
packets: 118284
},
{
bytes: 15505499,
ordered_by: 'packets',
packets: 33776,
source_interface: '4.5.6.7:31'
},
{
bytes: 44638204,
elements: 13,
packets: 118284,
source_interface: 'total'
},
{
bytes: 44638204,
ordered_by: 'packets',
packets: 118284,
source_country: 'xx'
},
{
bytes: 44638204,
ordered_by: 'bytes',
packets: 118284,
source_country: 'xx'
},
{
bytes: 44638204,
elements: 1,
packets: 118284,
source_country: 'total'
},
{
bytes: 44638204,
destination_country: 'xx',
ordered_by: 'packets',
packets: 118284
},
{
bytes: 44638204,
destination_country: 'xx',
ordered_by: 'bytes',
packets: 118284
},
{
bytes: 44638204,
destination_country: 'total',
elements: 1,
packets: 118284
},
{
bytes: 220,
destination_interface: '6.7.8.8:15811387',
ordered_by: 'bytes',
packets: 3
},
{
bytes: 44638204,
destination_interface: 'total',
elements: 15,
packets: 118284
},
{
bytes: 44638204,
device_address: '2.3.4.5',
ordered_by: 'packets',
packets: 118284
},
{
bytes: 44638204,
device_address: '3.4.5.6',
ordered_by: 'bytes',
packets: 118284
},
{
bytes: 44638204,
device_address: 'total',
elements: 1,
packets: 118284
}
],
host_group: 'global',
sample_end: '2024-09-05T16:00:23Z',
sample_start: '2024-09-05T15:59:23Z'
}
]
Please note that after listing 100 values elements for particular field (for example ASN):
{
bytes: 29768343,
destination_asn: '269872',
ordered_by: 'packets',
packets: 68756
},
Each field is ordered by packets and bytes and you can find sorting value this way in section “ordered_by”:
{
bytes: 44638204,
device_address: '2.3.4.5',
ordered_by: 'packets',
packets: 118284
},
{
bytes: 44638204,
device_address: '3.4.5.6',
ordered_by: 'bytes',
packets: 118284
},
We add total field which carries total amount of packets and bytes for this field:
{
bytes: 44638204,
destination_asn: 'total',
elements: 174,
packets: 118284
},
To debug performance of this capability we have dedicated system counter:
sudo fcli show system_counters|grep hostgroup_traffic_samples hostgroup_traffic_samples_calculation_time 0.000000
After that you will be able to request max talkers list via API. You can do request to retrieve all records this way:
curl -X GET -u admin:password http://127.0.0.1:10007/hostgroup_traffic_samples
To retrieve records from specific date please use another query:
curl -X GET -u admin:password http://127.0.0.1:10007/hostgroup_traffic_samples/2022-02-01T00:58:40Z
FastNetMon has logic to remove these entries from MongoDB automatically every 15 minutes. Before version 2.0.347 we run cleanup logic once per hour.
If you run query and API responds error 500 like this:
[martini]2024/04/12 15:45:56 Completed GET /hostgroup_traffic_samples/2023-04-12T31:44:38Z 500 Internal Server Error in 7.135398ms [martini]2024/04/12 15:45:56 Show function returned error: We do not have any samples for your query
Then it means that for some reasons FastNetMon was not able to calculate traffic profile and store them to MongoDB. Very likely you will find details at /var/log/fastnetmon/fastnetmon.log.
Please note that FastNetMon itself can calculate bandwidth for specific entities like ASNs using it’s own real time bandwidth calculation protocol and such calculations cannot be compare to bandwidth calculated using data from hostgroup samples as bandwidth calculated from amount of transferred traffic is inaccurate by it’s own nature.
Timeout issues when large number of hostgroups presents
In cases if you have large number of hostgroups (100+) you may face issues like this:
curl -x GET -k -u admin:admin https://1.2.3.4/hostgroup_traffic_samples
{"success":false, "error_text": "Could not retrieve configuration from database context deadline exceeded"
This issue means that due to very large amount of data in MongoDB fcli cancels query as it takes too long.
To address this issue you may consider increasing timeout for heavy operations (it applies for hostgroups_max_talkers, hostgroup_traffic_stats, attack_traffic_stats) in /etc/fastnetmon/fastnetmon.conf:
{
"mongodb_timeout_heavy": 95
}
After that this issue will disappear. Please note that this flag was introduced in 2.0.363 and you need to upgrade to this version to use this capability