Please note that this capability is part of a partner-only integration feature set, and we strongly advise against using it unless you have discussed it with our engineering team.
This logic can be changed or removed without any further notice, and it is not a part of FastNetMon's backward compatibility guarantee.
To enable per-hostgroup traffic profile collection logic, you will need to enable these configuration options:
sudo fcli set main generate_hostgroup_traffic_samples true sudo fcli set main generate_hostgroup_traffic_samples_delay 60 sudo fcli commit
After that, FastNetMon will calculate the amount of traffic with drop-down by type of traffic for each per_host hostgroup you have in configuration and will store them to MongoDB in the collection: hostgroup_traffic_stats. It will do it every generate_hostgroup_traffic_samples_delay seconds.
FastNetMon exports the amount of transferred traffic using the following fields:
- source_ip
- destination_ip
- protocol
- ttl
- packet_length
- tcp_flags
- source_port
- destination_port
- source_country
- destination_country
- source_asn
- destination_asn
- source_interface
- destination_interface
- device_address
Please note that this capability depends on a correctly configured traffic buffer, and if you forget to configure it, you will see the following error in /var/log/fastnetmon/fastnetmon.log
Please enable traffic buffer to generate hostgroup traffic profiles
To get them in MongoDB you can use the following query:
db.hostgroup_traffic_stats.find({})
Example report looks this way:
[
{
_id: ObjectId('66d9d598dc1c421e770e3426'),
fields: [
{
bytes: 5760183,
ordered_by: 'packets',
packets: 39064,
source_ip: '1.2.3.4'
},
....
{
bytes: 44638204,
elements: 1984,
packets: 118284,
source_ip: 'total'
},
...
{
bytes: 15005,
destination_ip: '2.3.4.5',
ordered_by: 'bytes',
packets: 44
},
{
bytes: 44638204,
destination_ip: 'total',
elements: 1449,
packets: 118284
},
{
bytes: 25518912,
ordered_by: 'packets',
packets: 87422,
protocol: '6'
},
{
bytes: 18890272,
ordered_by: 'packets',
packets: 27083,
protocol: '17'
},
{
bytes: 218682,
ordered_by: 'packets',
packets: 3712,
protocol: '1'
},
{
bytes: 4600,
ordered_by: 'packets',
packets: 37,
protocol: '50'
},
{
bytes: 5738,
ordered_by: 'packets',
packets: 30,
protocol: '47'
},
{
bytes: 25518912,
ordered_by: 'bytes',
packets: 87422,
protocol: '6'
},
{
bytes: 18890272,
ordered_by: 'bytes',
packets: 27083,
protocol: '17'
},
{
bytes: 218682,
ordered_by: 'bytes',
packets: 3712,
protocol: '1'
},
{ bytes: 5738, ordered_by: 'bytes', packets: 30, protocol: '47' },
{ bytes: 4600, ordered_by: 'bytes', packets: 37, protocol: '50' },
{
bytes: 44638204,
elements: 5,
packets: 118284,
protocol: 'total'
},
{
bytes: 44638204,
ordered_by: 'packets',
packets: 118284,
ttl: '0'
},
{
bytes: 44638204,
ordered_by: 'bytes',
packets: 118284,
ttl: '0'
},
{ bytes: 44638204, elements: 1, packets: 118284, ttl: 'total' },
{
bytes: 43249862,
ordered_by: 'packets',
packet_length: 'jumbo_packets',
packets: 103514
},
{
bytes: 732127,
ordered_by: 'packets',
packet_length: '0_150',
packets: 10097
},
{
bytes: 224152,
ordered_by: 'packets',
packet_length: '151_300',
packets: 1655
},
{
bytes: 115167,
ordered_by: 'packets',
packet_length: '301_450',
packets: 747
},
{
bytes: 75581,
ordered_by: 'packets',
packet_length: '751_900',
packets: 725
},
{
bytes: 50099,
ordered_by: 'packets',
packet_length: '1051_1200',
packets: 391
},
{
bytes: 47632,
ordered_by: 'packets',
packet_length: '601_750',
packets: 361
},
{
bytes: 23177,
ordered_by: 'packets',
packet_length: '1201_1350',
packets: 225
},
{
bytes: 70347,
ordered_by: 'packets',
packet_length: '451_600',
packets: 222
},
{
bytes: 26153,
ordered_by: 'packets',
packet_length: '901_1050',
packets: 212
},
{
bytes: 23907,
ordered_by: 'packets',
packet_length: '1351_1500',
packets: 135
},
{
bytes: 43249862,
ordered_by: 'bytes',
packet_length: 'jumbo_packets',
packets: 103514
},
{
bytes: 732127,
ordered_by: 'bytes',
packet_length: '0_150',
packets: 10097
},
{
bytes: 224152,
ordered_by: 'bytes',
packet_length: '151_300',
packets: 1655
},
{
bytes: 115167,
ordered_by: 'bytes',
packet_length: '301_450',
packets: 747
},
{
bytes: 75581,
ordered_by: 'bytes',
packet_length: '751_900',
packets: 725
},
{
bytes: 70347,
ordered_by: 'bytes',
packet_length: '451_600',
packets: 222
},
{
bytes: 50099,
ordered_by: 'bytes',
packet_length: '1051_1200',
packets: 391
},
{
bytes: 47632,
ordered_by: 'bytes',
packet_length: '601_750',
packets: 361
},
{
bytes: 26153,
ordered_by: 'bytes',
packet_length: '901_1050',
packets: 212
},
{
bytes: 23907,
ordered_by: 'bytes',
packet_length: '1351_1500',
packets: 135
},
{
bytes: 23177,
ordered_by: 'bytes',
packet_length: '1201_1350',
packets: 225
},
{
bytes: 44638204,
elements: 11,
packet_length: 'total',
packets: 118284
},
{
bytes: 9807668,
ordered_by: 'packets',
packets: 35255,
tcp_flags: 'psh,ack'
},
{
bytes: 10749258,
ordered_by: 'packets',
packets: 30799,
tcp_flags: 'ack'
},
{
bytes: 1486870,
ordered_by: 'packets',
packets: 14286,
tcp_flags: 'syn'
},
{
bytes: 3327593,
ordered_by: 'packets',
packets: 5086,
tcp_flags: 'syn,ack'
},
...
{
bytes: 25518912,
elements: 253,
packets: 87422,
tcp_flags: 'total'
},
{
bytes: 17114707,
ordered_by: 'packets',
packets: 34290,
source_port: '8291'
},
...
{
bytes: 44409184,
elements: 2703,
packets: 114505,
source_port: 'total'
},
{
bytes: 3937222,
destination_port: '8291',
ordered_by: 'packets',
packets: 31008
},
...
{
bytes: 44409184,
destination_port: 'total',
elements: 4317,
packets: 114505
},
{
bytes: 14869861,
ordered_by: 'packets',
packets: 49528,
source_asn: '269872'
},
{
bytes: 44638204,
elements: 252,
packets: 118284,
source_asn: 'total'
},
{
bytes: 29768343,
destination_asn: '269872',
ordered_by: 'packets',
packets: 68756
},
...
{
bytes: 44638204,
destination_asn: 'total',
elements: 174,
packets: 118284
},
{
bytes: 15505499,
ordered_by: 'packets',
packets: 33776,
source_interface: '4.5.6.7:31'
},
{
bytes: 44638204,
elements: 13,
packets: 118284,
source_interface: 'total'
},
{
bytes: 44638204,
ordered_by: 'packets',
packets: 118284,
source_country: 'xx'
},
{
bytes: 44638204,
ordered_by: 'bytes',
packets: 118284,
source_country: 'xx'
},
{
bytes: 44638204,
elements: 1,
packets: 118284,
source_country: 'total'
},
{
bytes: 44638204,
destination_country: 'xx',
ordered_by: 'packets',
packets: 118284
},
{
bytes: 44638204,
destination_country: 'xx',
ordered_by: 'bytes',
packets: 118284
},
{
bytes: 44638204,
destination_country: 'total',
elements: 1,
packets: 118284
},
{
bytes: 220,
destination_interface: '6.7.8.8:15811387',
ordered_by: 'bytes',
packets: 3
},
{
bytes: 44638204,
destination_interface: 'total',
elements: 15,
packets: 118284
},
{
bytes: 44638204,
device_address: '2.3.4.5',
ordered_by: 'packets',
packets: 118284
},
{
bytes: 44638204,
device_address: '3.4.5.6',
ordered_by: 'bytes',
packets: 118284
},
{
bytes: 44638204,
device_address: 'total',
elements: 1,
packets: 118284
}
],
host_group: 'global',
sample_end: '2024-09-05T16:00:23Z',
sample_start: '2024-09-05T15:59:23Z'
}
]
Please note that after listing 100 values elements for a particular field (for example, ASN):
{
bytes: 29768343,
destination_asn: '269872',
ordered_by: 'packets',
packets: 68756
},
Each field is ordered by packets and bytes, and you can findthe sorting value this way in the section "ordered_by":
{
bytes: 44638204,
device_address: '2.3.4.5',
ordered_by: 'packets',
packets: 118284
},
{
bytes: 44638204,
device_address: '3.4.5.6',
ordered_by: 'bytes',
packets: 118284
},
We add a total field which carries the total amount of packets and bytes for this field:
{
bytes: 44638204,
destination_asn: 'total',
elements: 174,
packets: 118284
},
To debug the performance of this capability, we have a dedicated system counter:
sudo fcli show system_counters|grep hostgroup_traffic_samples hostgroup_traffic_samples_calculation_time 0.000000
After that, you will be able to request max talkers list via API. You can make a request to retrieve all records this way:
curl -X GET -u admin:password https://127.0.0.1:10007/hostgroup_traffic_samples
To retrieve records from a specific date, please use another query:
curl -X GET -u admin:password https://127.0.0.1:10007/hostgroup_traffic_samples/2022-02-01T00:58:40Z
FastNetMon has logic to remove these entries from MongoDB automatically every 15 minutes. Before version 2.0.347 we ran cleanup logic once per hour.
If you run a query and the API responds with error 500 as follows:
[martini]2024/04/12 15:45:56 Completed GET /hostgroup_traffic_samples/2023-04-12T31:44:38Z 500 Internal Server Error in 7.135398ms [martini]2024/04/12 15:45:56 Show function returned error: We do not have any samples for your query
This means that for some reason FastNetMon was unable to calculate traffic profiles and store them in MongoDB. You will likely find details at /var/log/fastnetmon/fastnetmon.log.
Please note that FastNetMon itself can calculate bandwidth for specific entities, such as ASNs, using its own real-time bandwidth calculation protocol, and such calculations cannot be compared to bandwidth calculated using data from hostgroup samples, as bandwidth calculated from the amount of transferred traffic is inaccurate by its own nature.
Timeout issues with a large number of hostgroups
In cases you have a large number of hostgroups (100+), you may face issues such as following:
curl -x GET -k -u admin:admin https://1.2.3.4/hostgroup_traffic_samples
{"success":false, "error_text": "Could not retrieve configuration from database context deadline exceeded"
This error means that due to a very large amount of data in MongoDB, fcli cancels the query as it takes too long.
To address this issue, you may consider increasing the timeout for heavy operations (it applies to hostgroups_max_talkers, hostgroup_traffic_stats, attack_traffic_stats) in /etc/fastnetmon/fastnetmon.conf:
{
"mongodb_timeout_heavy": 95
}
After that, this issue will disappear. Please note that this flag was introduced in 2.0.363, and you need to upgrade to this version to use this capability.