Changes

Version 2.0.250

• Fixed bugs in af packet and xdp ng gre enabled parser
• Added more counters for AF_PACKET logic and fixes bug for XDP and AF PACKET for new generation parser
• Expanded logic to disable offload for vmxnet3
• Added logic to automatically enable InfluxDB export from visual installer
• Added logic to set secure permissions for htpasswd file
• We need to disable using force yes as it may break things
• Added mirror_af_external_packet_sampling and mirror_external_af_packet_sampling_rate values for AF PACKET
• Added flag to extract tunnel traffic for af packet mode
• Added NG parser flag for AF_PACKET
• Added NG parser flag for XDP

Version 2.0.249

• Added logic to avoid partial deserialization for IPFIX
• Added intermediate step for template serialize for Netflow 9 to avoid issues with deserializer
• Reworked Netflow and IPFIX template cache logic
• Added blackhole v6 cleanup for Debian and Ubuntu

Version 2.0.248

• Added logic to parse GRE from ng parser
• Added logic to identify GRE and ESP protocols properly

Version 2.0.247

• Rename IPFIX file to typo-free version
• Added support for unban when finished for IPv6

Version 2.0.246

• Migrated MongoDB to explicit copy of session in attempt to address issue: ‘database closed explicitly’
• Disabled trace logic by default for Web API, only print errors

Version 2.0.245

• Added option to specify custom port for BGP: gobgp_bgp_listen_port

Version 2.0.244

• Fixed seg fault when both web hook and script were enabled
• Propagate license error to top level call of set renew_license
• Added logic to remove cache entries between upgrades on Debian and Ubuntu

Version 2.0.243

• Added auth support for Grafana alerts

Version 2.0.242

• Added flow spec next hop logic support: gobgp_flow_spec_next_hop_ipv4 and gobgp_flow_spec_next_hop_ipv6

Version 2.0.241

• Added logic to exclude traffic by vlan
• Added logic to enable SSL on Ubuntu and Debian

Version 2.0.240

• Upgraded new GoBGP from 2.16 to 2.27

Version 2.0.239

• Implemented BGP flow spec and BGP for new GoBGP integration

Version 2.0.238

• Added simple packet dump for flow spec dumps

Version 2.0.237

• Added packet_dump_detailed for flow spec dumps

Version 2.0.236

• Added IP fragments threshold

Version 2.0.235

• Fixed bug with rate limit encoding in flow spec

Version 2.0.234

• Added complete support for ESP and GRE protocols

Version 2.0.233

• Full implementation of traffic rules for fcli
• Fixed obvious bugs in mongo based autcompletion
• Unified auto completion and eliminated sort logic for names of field

Version 2.0.232

• Reworked configuration management in fcli

Version 2.0.231

• Added logic to filter out IPv6 announces for IPv4 ASN lookup tree

Version 2.0.230

• Added flag to suppress uuid in flow spec when we just list rules for fcli
• Added UUID of rule for flow spec callback
• Added logic to parse sFlow IPv4 headers without Ethernet headers
• Added per protocol counters for sFlow headers

Version 2.0.229

• Added logic to populate thresholds used to detect attack
• Exposed attack_detection_threshold to JSON

Version 2.0.228

• Added logic to generate UUID for manually banned hosts
• Enabled traffic_db by default
• Added new SQL schema for TTL based retention
• Added compression for Grafana to improve Nginx performance
• Moved away from stringstream and additional copy of memory to improve sFlow performance
• Added new faster version of IP conversion logic

Version 2.0.227

• Added logic to distinguish new sampling rate for IPFIX and update of sampling rate

Version 2.0.226

• Fixed missing outgoing traffic for ASNs
• Extracted FNM type mapping and deprecated fields into function
• Upgrade golang version
• Switched intaller to go mod
• Migrated fcli to go mod
• Migrated fastnetmon_client to go mod
• Added etcd to distribution package

Version 2.0.225

• Fixed bug with flow calculatiuons in flexible counters mode
• Removed ASN lookup from traffic_db, please enable asn_lookup on FastNetMon side instead
• Added logic to lookup ASN for IPv6 traffic
• Added fcli commands: asn_counters_v4 and asn_counters_v6
• Added Clickhouse export for ASNs
• Added ASN export to InfluxDB
• Extracted IPv6 logic into separate function

Version 2.0.224

• Added logic to decapsulate sflow tunnels: sflow_extract_tunnel_traffic

Version 2.0.223

• Added IPv6 incoming announces listing

Version 2.0.222

• Enabled Netflow lite support: netflow_v9_lite

Version 2.0.221

• Added logic to re-classify IPv6 packets as IPv4

Version 2.0.220

• Reduced amount of data copies for flow spec announces
• Added logic to generated UUID for flow spec in GoBGP logic
• Switched to manual uuid generation for flow spec announces
• Removed uuids for unicast v4 and v6 announces
• Unified attack_details_t and banlist_item_t, we do not need latter, it just adds uselss complexity
• Fixed gcc warning about Flow Spec rate-limit little / big endian covnersion
• Added functions to expose number of packets per device
• Dropped plenty of deprecated conf options

Version 2.0.219

• Enabled IPv6 traffic processing by default in configuration
• Fixed 64 bit overflow error: Cannot parse value 16101149398 for metric InDatagrams
• Added sflow_read_packet_length_from_ip_header to read packet length from ip packet instead of sFlow

Version 2.0.218

• Added logic to stop processing when we read all flow sets. Fixed bug with Netflow with garbadge on the end of packet
• Reduced error duplication for IPFIX
• Added support for RHEL

Version 2.0.217

• Added working support for flow counters for flexible counters
• Unified logic for flow tracking
• Covered access to SubnetVectorMapFlow with mutex

Version 2.0.216

• Reduced logging from hot path
• Added conf option netflow_rx_queue_overflow_monitoring for recvmsg mode

Version 2.0.215

• Added staticstics for global udp packet loss
• Extracted banlist logic into separate module

Version 2.0.214

• Added conf option netflow_socket_read_mode which can be set to: recvmsg or recvfrom to control socket mode. recvmsg offers monitoring for number of drops on socket
• Added logic to dump default sflow and netflow buffer size

Version 2.0.213

• Added detailed logging for license validation errors
• Suppressed http logging for InfluxDB in installer code
• Added logic to skip firewalld configuration if we do not have it on CentOS

Version 2.0.212

• Fixed overflow for json serializer: it used signed 32 bit value for 64 bit integers: 2629580 pps 22169 mbps

Version 2.0.211

• Added flag for instant ban mode without packet collection
• Added gobgp_flow_spec_v6_announces, gobgp_flow_spec_v6_default_action, gobgp_flow_spec_v6_rate_limit_value to control flow spec for IPv6

Version 2.0.210

• Added logic to retrieve IPv6 host counters for host

Version 2.0.209

• Disabled repeatative logs which spam log files in debug mode
• Added metric about number of not parsed packets

Version 2.0.208

• Moved Clickhouse metrics to the separate module
• Graphite extracted into separate module
• Graphite extracted into separate module

Version 2.0.207

• Added options to override sampling rate for Netflow on router basis for ipfix_per_router_sampling_rate, netflow_v9_per_router_sampling_rate, netflow_v5_per_router_sampling_rate

Version 2.0.206

• Added support for IPFIX sampling rates
• Added newer field which may carry IPFIX sampling rate for Cisco NCS devices
• Added fcli icommand to read sampling rate for ipfix devices: show ipfix_sampling_rates
• Added sanity check logic for IPFIX options data
• Added sanity check logic for Netflow v9 data
• Added counter to track number of IPFIX option templates
• Added logic to track case when flow duration is negative
• Added logic to decode reasons of flow timeout on Juniper for IPFIX
• Fixed issue which caused FastNetMon autostart removal after upgrade on CentOS

Version 2.0.205

• Implemented option to override number of workers for AF_PACKET manually: mirror_af_packet_workers_number_override, mirror_af_packet_workers_number
• Set number of AF_PACKET workers to number of NIC queues
• Added logic to parse FORWARDING _STATUS for Netflow v9

Version 2.0.204

• Disabled automatic enablement for pcap dumps, it’s not required in modern detection engine
• Removed experimental logic to divide packet on duration
• Made code more compliant with C++ specification
• Removed old code for Netflow v1 and v7 support
• Added alternative ways to declare bool values in cli / api
• Added logic to disabled offload automatically for AF_PACKET
• Added logic to disable offload on XL710 Intel

Version 2.0.203

• Added input and output interfaces for sFlow

Version 2.0.202

• Added logic to generate ASN mapping list from fill_dictionaries

Version 2.0.201

• Added SSL support for API
• Added duplicate network prevention for total and per host IPv4 and IPv6 hostgroups
• Sync Grafana configuration with upstream
• Sync InfluxDB configuration with upstream
• Aded logic to override Netflow v5 sampling rate: netflow_v5_custom_sampling_ratio_enable and netflow_v5_sampling_ratio
• Added CentOS support for Grafana, InfluxDB, Clickhouse, Nginx
• Disabled core dumps by default
• Enabled CentOS support in installer

Version 2.0.200

• Enabled MongoDB authentication in Docker build
• Added complete ARM64 support for Ubuntu 16.04, 18.04 and 20.04

Version 2.0.199

• Added flag Gobgp_do_not_manage_daemon to avoid attempts to restart GoBGP
• Added flags gobgp_api_host and gobgp_api_port to configure data for GoBGP connection
• Added logic to disable AppArmor for msmtp and added more details about msmtp operations
• Switched to new gRPC
• Added opton flow_spec_ignore_do_not_fragment_flag to address Arista issues

Version 2.0.198

• Added gobgp_communities_subnet_ipv4 to specify multiple subnet level communities
• Added new option gobgp_communities_host_ipv4 to specify multiple communities for IPv4 host announces
• Added metric sflow_raw_udp_packets_received and sflow_udp_receive_errors to track UDP errors in sFlow mode
• Added support for Debian in graphic stack
• Added support for Ubuntu 20.04
• Added Debian support for FastNetMon

Version 2.0.197

• Switched to custom version of OpenSSL
• Fixed API support on Google Cloud / Firestore
• Added Clickhouse error ignore if we cannot start it due to missing SSE42
• Added new flag mongo_store_attack_information to address crashes on GCE

Version 2.0.196

• Enabled export into traffic_db for traffic received from tera_flow plugin
• Fixed bug with deprecated fields on FireStore backend
• Added metric total_flowspec_filtered_bytes to count number of bytes filtered by flow spec rules

Version 2.0.195

• Added option to ignore IP flags: flow_spec_do_not_process_ip_fragmentation_flags_field

Version 2.0.194

• Added source of attack detection for remote blocks
• Added conf option flow_spec_do_not_process_tcp_flags_field

Version 2.0.193

• Upgrade JSON library
• Moved lots of error messages to DEBUG level

Version 2.0.192

• Added syslog status messages

Version 2.0.191

• Added bps and pps rates for incoming and outgoing for syslog attack alerts

Version 2.0.190

• Added option to make syslog alerts

Version 2.0.189

• Disabled flow spec call if we do not have any rules
• Added vlan for ng parser
• Fixed bug with too big packet buffer size in IPv6 mode
• Exported parent_host_group flag for per host json callbacks
• Fixed misnamed collect_attack_pcap_dumps and collect_simple_attack_dumps to their original meaning

Version 2.0.188

• Fixed out of bound memory write in IPFIX code which break packet counter
• Added logic to handle crashes when malformed data was sent to tera_flow port
• Fixed bug caused by flexible hash clenup logic and when option unban_only_when_finished was set
• Added conf options override_internal_traffic_as_incoming and override_internal_traffic_as_outgoing to alter logic for internal traffic

Version 2.0.187

• Added Radware plugin fcli management

Version 2.0.186

• Added flow per second rate

Version 2.0.185

• gRPC and protobuf libraries upgrade

Version 2.0.184

• Added TCP flag support for flow spec mode for Netflow capture

Version 2.0.183

• Upgraded all libraries
• Added options netflow_ignore_long_duration_flow_enable and netflow_long_duration_flow_limit and basic implementation of duration filter

Version 2.0.182

• Added fcli autocompletion for bgp peers
• Added support for context help for bgp and hostgroup
• Updated cli library
• Added logic to read flow spec announces from disk
• Added option keep_flow_spec_announces_during_restart and added on-disk sync when we remove flos spec announce

Version 2.0.181

• API: Added logic to retry connection to MongoDB 30 times with 5 second delay during initial startup
• Added confirmation for create_configuration command in interactive mode
• Allowed symbol – and uppercase letters in hostgroup and bgp peer names
• Added logic to hide any passwords in configuration
• Added agent IP information to details json packet dump
• Added ASN and interface numbers in details JSON dump of packet

Version 2.0.180

• Improved autocompletion for hostgroups
• Fixed autocompletion artefacts (duplication) with many completion matches

Version 2.0.179

• Improved blackhole persistency logic

Version 2.0.178

• Added logic to immediately sync blocked IPv4 and IPv6 hosts with on disk dump for graceful restarts

Version 2.0.177

• Added support for parent hostgroups to use networks list from another group

Version 2.0.176

• Added IPv6 support for total hostgroups
• Added IPv6 support for per-host hostgroups
• Added support logic to parse ASN from sFlow

Version 2.0.175

• We’ve deperecated flag netflow_ipv6 and now you need only process_ipv6_traffic

Version 2.0.174

• Added option to autoamtically cleanup empty counters for IPv4 and IPv6 flexible counters. Significantly reduces CPU usage

Version 2.0.173

• Added TCP SYN bandwidth and packet per second thresholds
• Enabled IPv6 counter even if IPv6 processing is disabled completely

Version 2.0.172

• Added support for offline licenses on machines without default route

Version 2.0.171

• Added flows per second export to tracking server for new custom license types
• Added counters about per header and per flow speed for system_counters

Version 2.0.170

• Added logic to multiply packet rate on sampling rate in af packet mode
• Fixed extremely old bug with traffic conversion to mbits it was cause of 5% traffic difference

Version 2.0.169

• Improved build process

Version 2.0.168

• Implemented option to export IPv6 counters to InfluxDB

Version 2.0.167

• Added native Prometheus support for system counters
• Improved Netflow error reporting in log

Version 2.0.166

• Enabled flexible_traffic_calculation by default

Version 2.0.165

• Added options gobgp_announce_whole_subnet_force_custom_prefix_length and gobgp_announce_whole_subnet_custom_prefix_length to override length for subnet announces
• Added option to install Docker on Ubuntu 18.04
• Added options gobgp_announce_whole_subnet_force_custom_ipv6_prefix_length and gobgp_announce_whole_subnet_custom_ipv6_prefix_length to override IPv6 announces
• Improved logging for orphaned buckets cleanup
• Unified IPv6 and IPv4 bucket cleanup function
• Added debug logging for packet collection
• Implemented batch processing for IPv4 and IPv6 bans. Previously, FastNetMon was able to run single ban action per ban function run (once per second). In this release it can ban any number of hosts each ban function call. It makes FastNetMon much more efficient agains attacks towards big number of hosts

Version 2.0.164

• Added option netflow_ignore_sampling_rate_from_device to ignore Netflow v9 sampling rate announcements from device

Version 2.0.163

• Added per protocol traffic counters for metrics export to Clickhouse: clickhouse_metrics_per_protocol_counters

Version 2.0.162

• Added code to unban hostgroups automatically
• Added attack notification pipeline for manual unblocks for hostgroups

Version 2.0.161

• Added networks which belong to hostgroup to per-hostgroup callbacks
• Added scope to distinguish per-host and per-hostgroup callbacks

Version 2.0.160

• Suppressed log messages about InfluxDB and switched them to counters
• Added option to build total hostgroups from per-host: build_total_hostgroups_from_per_host_hostgroups

Version 2.0.159

• Implemented logic to export per protocol counters for hostgroups

Version 2.0.158

• Added option to suppress automatic flow spec withdrawal: do_not_withdraw_flow_spec_announces_on_restart
• Fixed bug with mongo session in fcli

Version 2.0.157

• Added support to decode ASNs encoded as 2 byte values in IPFIX

Version 2.0.156

• Reduced fcli timeout from 10 to 3 seconds to offer better experience in offline mode
• Disabled increasing timeout during connection to MongoDB. Now we retry 10 times every 5 seconds

Version 2.0.155

• Aded option to suppress pid file check when pid_file path is empty

Version 2.0.154

• Added custom command to restart FNM on Docker
• Added custom path for FastNetMon’s log in Docker setup
• Added detailed error message to debug mmap issues
• Added offline mode for fcli to avoid scary errors during database init process

Version 2.0.153

• Added option to connect to MongoDB without authentication
• Disabled requirement for non empty Mongo password in fcli

Version 2.0.152

• Upgraded gRPC version in FastNetMon’s internal API
• Suppressed verbose logging from BGP parser code

Version 2.0.151

• Added support to export top remote talkers via fcli: show remote_host_counters

Version 2.0.150

• Added support to account total traffic usage per hostgroup
• Added threshold to run actions when total traffic for host group exceed limit
• Introduced notify_script_hostgroup_enabled and notify_script_hostgroup_path for per-hostgroup notify script calls
• Added flag enable_ban_hostgroup to control per hostgroup behaviour
• Implemented per hostgroup counters for InfluxDB
• Introduced optimized version of LPM lookup tree
• Decoupled code which reads hostgroups with code which creates all support structures

Version 2.0.149

• Added ASN lookup logic for IPv6 in traffic_db
• Unified IPv4 remote and IPv6 traffic counters
• Added IPv6 support for fill_dictionaries

Version 2.0.148

• Implemented complete GCE support

Version 2.0.147

• Added support for FireStore for hostgroups and bgp in fcli

Version 2.0.146

• Completely working code to withdraw remote blackhole announces
• Exported information about license status to FCLI
• Added code which will trigger all callbacks for manually added flow spec rules

Version 2.0.145

• Added option to add custom tags to InfluxDB metrics: influxdb_custom_tags, influxdb_tag_name, influxdb_tag_value
• Added option to control partitioner between Kafka partitions: influxdb_kafka_partitioner
• Added tracking for number of InfluxDB messages written to influxdb for system_counters

Version 2.0.144

• Implemented export to InfluxDB over Kafka queue system
• Fixed traffic_db.conf parser issues with empty keys

Version 2.0.143

• Added Clickhouse bandwidth export for flexible counters mode

Version 2.0.142

• Added system_counters hosts_hash_size_ipv6 and hosts_hash_load_factor_ipv6 for IPv6 performance debugging
• Exposed debug options for flexible counters: hosts_hash_load_factor and hosts_hash_size
• Added support for flexible counters which avoid memory allocation for hosts without traffic and sparse networks
• FastNetMon will execute license check after checking for duplicate process in memory

Version 2.0.141

• Enabled simple packet dump in email by default
• Added command for installer to set password for Grafana from metdata service on GCE (sudo ./installer -set_visual_passwords_gce_metadata)
• Added logic to use per VM configuration storage on GCE / Firebase

Version 2.0.140

• Reduced amount of memory copy calls in Netflow code
• Added counter to calculate number of Netflow v5 flows explicitly: netflow_v5_total_flows
• Added conf option netflow_multi_thread_processing to control multi thread mode for Netflow
• Added option to control number of working threads per port for Netflow: netflow_threads_per_port

Version 2.0.139

• Fixed bug for secure version of http client
• Added support for reading configuration from Google FireStore on GCE in FastNetMon

Version 2.0.138

• Added support for storing information in Google Firebase from fcli
• Added command to renew license automatically from fcli: sudo fcli set renew_licens

Version 2.0.137

• Added 4 byte encoding support for IPFIX plugin to decode flow starts and ends

Version 2.0.136

• Switched connection to license server to port 443 for better firewall compatibility

Version 2.0.135

• Added ability to set number of hosts in output for fastnetmon_client
• Added explicit code to process negative traffic recalculation delays
• Added return code processing for notify script with arguments
• Disabled ddos detection for outgoing traffic in default configuration to reduce false positive

Version 2.0.134

• Added packet_dump_detailed with per field information about packet dump to JSON notify scripts and web callbacks

Version 2.0.133

• Added option license_use_port_443 to switch to port 443 for all connections to license server

Version 2.0.132

• Added support for InfluxDB authorisation

Version 2.0.131

• Added new kind of whitelist which uses flow spec rules

Version 2.0.130

• Passed client ip address from sFlow to Clickhouse

Version 2.0.129

• Increased timeout for waiting Clickhouse from 3*5 to 5*15
• Added notifications about BGP Flow spec mitigations in Grafana

Version 2.0.128

• Added function to retrieve number of queues available on NIC
• Added XDP_ZEROCOPY support for AF_XDP (not implemented yet)
• Implemented single_remote_host_counters from fcli side
• Migrated ARM64 build to Mongo 4.1. It has native repos for Ubuntu Bionic

Version 2.0.127

• Added option to decapsulate external GRE tunnels: xdp_extract_tunnel_traffic

Version 2.0.126

• Added counter for packets unparsed by XDP
• Added option to switch interface into promisc for XDP

Version 2.0.125

• Enabled influxdb_export_system_counters by default.
• Changed influxdb host description: we allow domain names now
• Added DNS resolution code for InfluxDB export engine

Version 2.0.124

• Banned /0 IPv4 networks because customers can announces default gateway to FastNetMon using BGP integration
• Fixed code which should prevent allocating memory for really big networks
• Added new fields email_subject_blackhole_block, email_subject_blackhole_unblock, email_subject_partial_block

Version 2.0.123

• Implemented commit operation for web api. It can be trigered with PUT method
• Updated core libraries

Version 2.0.122

• Upgraded Mongo C and C++ libraries to current versions.
• Added fresh gRPC library for ARM64
• Added option keep_blocked_hosts_during_restart to control graceful reload of FastNetMon for blackholed hosts.
• Implemented logic to read blackholed hosts from disk
• Added option do_not_withdraw_unicast_announces_on_restart which disabled automatic BGP withdrawal

Version 2.0.121

• Added infinite loop prevention and limit for number of flowsets per packet

Version 2.0.120

• Added field packet_dump to JSON callback script and web hooks

Version 2.0.119

• Added debug message about field types in MongoDB
• Added new logic to parser to read integers encoded as 64 bit integers in MongoDB
• Regenerated parser for local_asn and remote_asn to support private 32 bit ASNs which exceed limit for 32 bit signed integer

Version 2.0.118

• Added native support for Grafana notifications about attacks

Version 2.0.117

• Fixed metric name from netflow_v9_duration_less_60_seconds to netflow_v9_duration_less_60_seconds
• Added automatic Clickhouse user configuration if we know it
• Switched to SHA 256 hashed password in Clickhouse configuration
• Fixed bug with incorrect name for Other Packets dashboard for total traffic
• Fixed bug with duplicated Incoming packets for per host traffic dashboard
• Added Netflow v9 dashboard into default package
• Nginx released native ARM64 packages for 18.04
• Added IPFIX duration histogram for system_counters

Version 2.0.116

• Added option remote_host_tracking to configure remote host tracking
• Added options enable_ban_remote_outgoing and enable_ban_remote_incoming to control remote blocks behaviour
• Added complete support for host groups remote_host_incoming/remote_host_outgoing for remote hosts
• Added gobgp_announce_remote_host option to announce blackholed remote hosts using BGP
• Added option gobgp_next_hop_remote_host to configure next hop for remote block hosts

Version 2.0.115

• Added plugin for AF_XDP support
• Added option force_native_mode_xdp to force native mode for XDP driver
• Added XDP stats
• Added poll backed processing for XDP mode: poll_mode_xdp. It significantly reduces load on CPU
• Disabled gobgp init file creation from installer. We do it from FastNetMon’s deb package
• Upgraded Grafana to 5.3.2.

Version 2.0.114

• Added experimental flag gobgp_modern_configuration_format to switch GoBGP to upstream format (1.33 compatible)
• Fixed bug with manual blackholes. Previously, all of them had 1970.1.1 date because we did not populate ban_time

Version 2.0.113

• Added option to disable milti-thread processing completely for AF_PACKET with option set main mirror_af_packet_disable_multithreading
• Replaced af_packet_ thread name to short afp_. We have limit for 16 symbols and it will improve performance profiling experience
• Added option to control fanout (load balancing mode) for AF_PACKET: mirror_af_packet_fanout_mode: cpu, hash, random, rollover, queue_mapping

Version 2.0.112

• HASH(0x2dc29b8)

Version 2.0.111

• Added command to configure mongodb initial configuration from fcli using fcli create_configuration

Version 2.0.110

• Fixed bug for Netflow v9 processing code which caused infinite loop after receiving malformed packet

Version 2.0.109

• Added ability to disable ban actions for incoming/outgoing traffic using: do_not_ban_outgoing and do_not_ban_incoming

Version 2.0.108

• Upgraded Clickhouse library to new version (fixes retries and timeouts)

Version 2.0.107

• Added option to switch connection tracking from flow to unique oppiste hosts tracking with sudo fcli set main connection_tracking_skip_ports

Version 2.0.106

• Implemented sudo fcli show netflow_sampling_rates to expose sampling rate information for Netflow v9 agents
• Added support for duration distribution for Netflow v5

Version 2.0.104

• Added IPv6 support for IPFIX

Version 2.0.104

• Fixed BGP Flow Spec validation bug for rules injected from API/fcli

Version 2.0.103

• Added fcli/API command: bgp_incoming_announces to expose received BGP announces from peer

Version 2.0.102

• Reintroduced Redis support

Version 2.0.101

• Added default fields for metricDate for Clickhouse metrics to avoid timezone issues
• Added automatic calculation for packetDate in traffic_db. We did it to eliminate timezone issues
• Added https support for apt in installer
• Enabled Ubuntu 18.04 by default
• Added support for 18.04 for graphic stack installer

Version 2.0.100

• Added agent ip address for Netflow v5
• Added agent ip address for Netflow v9 and IPFIX
• Added new field AgentIP address for Clickhouse
• Enabled support for 18.04 in installer and switched 18.04 to MongoDB 4.0

Version 2.0.99

• Added support for input and output interfaces for IPFIX
• Added src and dst ASN support for IPFIX
• Improved error reporting for Graphic stack installer
• Added support for Clickhouse server automatic install with: -install_traffic_persistency

Version 2.0.98

• Added ability to configure speed recalculation delay with option speed_calculation_delay. It’s quite useful for debugging and for customers with really huge networks

Version 2.0.97

• Added support for duration histogram for Netflow v9

Version 2.0.96

• Added variables netflow9_custom_sampling_rate_received and netflow9_sampling_rate_changes to track sampling rate learning for Netflow v9 better
• Added netflow9_options_packet_number counter to debug Netflow issues
• Initial support for viewing all availible options of fields in api

Version 2.0.95

• Added export for system counters to InfluxDB: set main influxdb_export_system_counters
• Added options for dpkg to suppress interactive reconfiguration attempts

Version 2.0.94

• Added IPv6 support for Netflow v9 code

Version 2.0.93

• Added https support for web hook handlers

Version 2.0.92

• Boost library upgrade to fix locale related issues and add new functions

Version 2.0.91

• Moved logging level reconfiguration close to toolkit run

Version 2.0.90

• Introduced configuration option to configure logging level: sudo fcli set main logging_level debug
• Fixed parser for flow spec detection enging for fragmented packets

Version 2.0.89

• Added support for flow_spec_do_not_process_length_field and flow_spec_do_not_process_source_address_field for flow spec mitigations in sFlow/AF_PACKET modes

Version 2.0.88

• Added configuration option flow_spec_do_not_process_source_address_field. It’s very useful to provide good level of aggregation for memcached/ssdp attack types

Version 2.0.87

• Added per day partitioning for Clickhouse metrics

Version 2.0.86

• Migrated to per day partitions for traffic_db/Clickhouse
• Removed dependencies on libpcap and libnuma from deb package

Version 2.0.85

• Added code for BPF sampling in AF_PACKET filter: mirror_af_packet_sampling_rate

Version 2.0.84

• Added ability to change traffic_db configuration using configuration file
• Added ARM64 compatibility
• luajit code upgarde for ARM64
• Added ARM64 version of installer tool
• Switched BGP daemon logging to stdout for systemd enabled distros

Version 2.0.83

• Improved compatibility with Ubuntu 18.04

Version 2.0.82

• Added support for accept BGP Flow Spec action type
• Unified upstart/systemd configuration for new Ubuntu distributions

Version 2.0.81

• Added ability to override Router ID for BGP peers: gobgp_router_id Mandatory for IPv6 only setup.
• Explicitly added local_address in BGP configuration

Version 2.0.80

• Enabled option to use IPv6 only peers

Version 2.0.79

• Implemented API and fcli to retrieve traffic for single host: fcli show single_host_counters 192.168.1.100

Version 2.0.78

• Added network for host into JSON and text/email notifictions
• Added protocol version to JSON and email/text notify scripts
• FastNetMon can populate host group properly for manually created blackholes
• Migrated MongoDB version 3.6

Version 2.0.77

• Added community support for IPv6 mode
• Added gobgp_community_host_ipv6 and gobgp_community_subnet_ipv6 to configure IPv6 independently
• Working IPv6 BGP integration
• Added configuration options for IPv6: gobgp_announce_host_ipv6 and gobgp_announce_whole_subnet_ipv6 to configure behaviour independently to IPv4
• Added field gobgp_next_hop_ipv6 to configure IPv6 next hop
• Implemented option to create IPv6 blackhole using cli
• Added IPv6 unicast and flow spec AFI for GoBGP.

Version 2.0.76

• Added basic ssh server into fcli: sudo -E SSH_SERVER_MODE=on ./fcli

Version 2.0.75

• Full IPv6 support
• Fixed whitelisting for IPv6
• Fixed segmentation faults when fastnetmon load IPv6 network into whitelist

Version 2.0.74

• Migrated to native Go interface code which creates fastnetmon user for MongoDB
• Installer does not remove old configiration. It just renames it
• Added autostart for BGP daemon
• Complete support for IPv6 for notify json script and web hook
• Implemented ability to show banned IPv6 address

Version 2.0.73

• Introduces IPv6 mode in fastnetmon_client -ipv6

Version 2.0.72

• Implemented ability for Netflow v9 to read input and output interfaces for packet from flow data
• Netflow v5. Added ability to read input and output port numbers from Netflow packets
• Added two new fields for Clickhouse schema: inputInterface and outputInterface

Version 2.0.71

• Fixed bug with ASN population code

Version 2.0.70

• Netflow plugin: implemented ability to read src and dst ASNs directly from Netflow v9 packet
• Traffic db will not try to fill ASN information if we already have non zero ASN from FastNetMon

Version 2.0.69

• Re-introduced fastnetmon_client toolkit to emulate old tool from FastNetMon Community
• Added symlink for fastnetmon_client to call without full path
• Introduced fake mode for fastnetmon_client using key z for example output

Version 2.0.68

• Implemented support for multiple interfaces in AF_PACKET
• Added new configuration option af_packet_read_packet_length_from_ip_header to read size from IP header instead of wire
• Added option for strict CPU affinity for AF_PACKET: afpacket_strict_cpu_affinity

Version 2.0.67

• Introduced complete support for licenses on grey IP addresses

Version 2.0.66

• More robust license check code

Version 2.0.65

• Complete web API
• Automatic creation for all Clickhouse tables
• Introduced ability to create database for Clickhouse metrics automatically
• Added correct check to handle unreachable Clickhouse for metrics export

Version 2.0.64

• Implemented option to export host counters using API

Version 2.0.63

• Enabled IPv6 by default in traffic_db
• Fixed bug with traffic_db. It crashed when we did not have Clickhouse running when traffic_db starts
• Added check for correctness of networks in cidr for for fcli

Version 2.0.62

• Introduced traffic_db_sampling_rate to configure sampling rate for AF_PACKET capture

Version 2.0.61

• Added automatic schema cration for Clickhouse

Version 2.0.60

• Fixed segmentation fault for fill_dictionaries: double memory free up

Version 2.0.59

• Enabled API gateway by default

Version 2.0.58

• Enabled code dump handlers for fill_dictionaries for debugging purposes

Version 2.0.57

• Introduced JSON mode for fcli: JSON_MODE=on sudo -E ./fcli show bgp
• Added JSON methods for almost all fcli commands

Version 2.0.56

• Enabled batch mode for Clickhouse by default
• Added automated deployment for all dashboards for Grafana
• Implemened an ability to create InfluxDB data source with API in Grafana automatically
• Improved InfluxDB installer to create database with proper retention

Version 2.0.55

• Added debug messages about orphaned buckets

Version 2.0.54

• Introduced support for md5 secured BGP sessions: md5_auth, md5_auth_password

Version 2.0.53

• Improved license validation code on grey IPs

Version 2.0.52

• Added options dump_all_traffic and dump_other_traffic to dump all/other traffic to log for debugging reasons
• Introduced new option sflow_use_new_generation_parser which could enable new packet parser for parsing sFlow packets

Version 2.0.51

• Added host group name to email/json notification about attack

Version 2.0.50

• Introduced an ability to skip export of host counters to reduce lod on InfluxDB: influxdb_skip_host_counters
• Added complete ignore to SIGHUP signal
• Added suport for Ubuntu 16.04 for graphic stack

Version 2.0.49

• Introuced per protocol counters export to InfluxDB: influxdb_per_protocol_counters

Version 2.0.48

• Added option to list all interfaces availible in system: show interfaces
• Fixed BGP daemon autostart unit

Version 2.0.47

• Introduced separate repo for Ubuntu Trusty Tahr

Version 2.0.46

• Made BGP Flow Spec valdiation configurable with configuration option: ‘flow_spec_execute_validation’
• Introduced option to enable/disable network aggregation to reduce number of networks: ‘aggregate_networks_list’
• Disabled configuration check for FastNetMon in systemd/Upstart unit

Version 2.0.45

• Improvement for users with huge networks lists: we do not add small networks already added into lookup table as part of big networks

Version 2.0.44

• Significant performance optimisation for users with big networks. 1.9 second => 0.7 second for /8 network
• Performance improvement: reduced lock contention for DDoS checks
• Added number of host groups to info log
• Performance improvement. 3.5 seconds -> 1.9 seconds for 1m hosts for speed recalculation. Replaced copy by value to copy by reference

Version 2.0.43

• Added fcli to PATH

Version 2.0.42

• Added initial support for systemd

Version 2.0.41

• Improved license validation code to ignore small differences in hardware configuration

Version 2.0.40

• Added option to add simple packet dump to email: email_notifications_add_simple_packet_dump

Version 2.0.39

• Introduced ASN lookup function for fcli: ‘show ip_asn 8.8.8.8’ you can use it to get ASN for any host

Version 2.0.38

• Improved compatibility with SMTP protocol in email notifications
• Added unique Linux thread names for all Netflow capture threads
• Fixed bug which caused negative source_id in template cache

Version 2.0.37

• Added new function for fcli to lookup host group for specified IP address: show ip_hostgroup 11.22.33.44
• Upgraded HTTP library to new version. We use it for web_hooks and InfluxDB metrics export

Version 2.0.36

• Added new configuration option flow_spec_do_not_process_length_field (main) to suppress any processing for length field. It’s useful in case when your device send incorrect IP length in Netflow/IPFIX

Version 2.0.35

• Introduced configuration option flow_spec_fragmentation_options_use_match_bit (main). You could use it to enable match field inside BGP Flow Spec announces. Some vendors require this flags
• Reworked code which handles BGP Flow Spec Fragmentation Flags

Version 2.0.34

• Added option flow_spec_tcp_options_use_match_bit (main) to enable match bit in BGP Flow Spec announces. Some vendors require this flags
• Traffic DB got an ability to create required databases/tables for traffic persistency automatically

Version 2.0.33

• Fixes bug with zero packetDate in traffic db
• In case of issues with connection with our traffic persistency backed FastNetMon could crash
• Hide email password from output of show main command

Version 2.0.32

• Added new plugin to read data in tera flow format from external sources (another FastNetMon instance).
• Implemented enhanced error handling for write_simple_packet
• Introduced filter to count full number of packets processed by process_packet function: total_simple_packets_processed

Version 2.0.31

• Fixed bug with active option for BGP peers. Now we handle it correctly
• Added support to store IPv6 address in tera flow format and inside tarffic persistency backed

Version 2.0.30

• Implemented batch inserter for traffic persistency backend (significantly faster than old implementation)
• Upgraded persistency database library integration. Fixed crash in case of connection issues with database

Version 2.0.29

• Introduced autconnect logic for traffic persistency backend

Version 2.0.28

• Rename process name fastnetmon to be unique line because it conflicts with Monit’s entity created for server itself
• Disabled graphite by default. If somebody wants it he could enable it manually
• Report incorrect value in field gobgp_flow_spec_default_action

Version 2.0.27

• Fixed whitelising option (main)

Version 2.0.26

• Implemented host_counters_v6 in fcli
• Implemented method to export top IPv6 talkers

Version 2.0.25

• Implemented full support for fcli command show sflow_sampling_rates. You could use it for sFlow sampling rate monitoring

Version 2.0.24

• Implemented an ability to specify number of hosts in output of show host_counters with env variable: HOST_COUNTERS_MAX_HOSTS

Version 2.0.23

• Introduced method license_hardware_data_reset to reset hardware assignment. You allowed to do it only three times

Version 2.0.22

• Fix bug with vendor specific implementation issues. When we have padding at the end of sFlow packet we drop all frames. Now we process all of them correctly

Version 2.0.21

• Introduced very nice optimization. Do not write zero elements to Clickhouse and InfluxDB
• We introduced show latest_fastnetmon_version for fcli to get latest version of toolkit from server
• Introduced ability to retrieve FastNetMon running version from fcli: show fastnetmon_version

Version 2.0.20

• Introduced ability to execute notify script when attack mitigated with blackhole (for ban and unban). Also, introduced ability to call it when attack was mitigatied with flow spec (script call on announce withdrawal is not supported yet). Please set notify_script_format to json to use this option
• Introduced variable to configure format for notify script. Now we support text and json formats

Version 2.0.19

• Allow small memory differences for license server
• Implemented network_counters_v6 in fcli
• Implemented ability to get sorted average counters for IPv6 subnets
• Fix bug in IPv6 get_packet_direction_ipv6. Now we could correctly detect outgoing traffic

Version 2.0.18

• Implemented support for extracing IPv6 total counters using API: show total_traffic_counters_v6
• Introduced special counter for non IPv4/IPv6 traffic: non_ip_packets.
• Introduced option to disable/enable IPv6 traffic processing explicitly: process_ipv6_traffic

Version 2.0.17

• Introduced ability to push network’s traffic to Clickhouse
• Introduced ability to configure Clickhouse for metrics using configuration interface

Version 2.0.16

• Fix bug in traffic presistency. Add correct traffic directions. Previously we had other everywhere

Version 2.0.15

• Print ASN mapping file version to log file
• Add special check that we could de-serialize data from ASN mapping table

Version 2.0.14

• Moved message about absence of template to DEBUG level because customer complaints
• Introduced additional log messages to debug Netflow templates delay

Version 2.0.13

• Fixed issue: netflow_host ignored in previous releases

Version 2.0.12

• Introduced ip_length field to track only IP packet total length (without Ethernet). We need it because BGP Flow Spec uses such representations

Version 2.0.11

• Since this commit we retrieve attack bandwidth information from time when atatck was detected. This commit fixes cases when customer report that attack report was below threshold
• Removed Average packet size for incoming traffic and Average packet size for outgoing traffic from attack details report
• Unified smtp notifications. Do not block thread with email process. Let’s fork additional threads for this task
• Introduced ability to remove packet length if we detected fragmentation attack. Implemented facility for rule optimisation
• Added requirement to run fill_dictionaries from root

Version 2.0.10

• Completely working ASN lookup for traffic database
• Introduced src_asn and dst_asn fields for traffic db and simple packet
• Implemented support for pushing data into Clickhouse

Version 2.0.9

• Added upstart file for tera-tsdb

Version 2.0.8

• Introduced ability to read MongoDB configuration options from special file in fcli
• Introduced option to specify custom mongodb port for fcli
• Moved configuration for mongodb for fcli to special file

Version 2.0.7

• Added api_port configuration variable
• Fixed bug in schema upgrade code inside CLI
• Install pwgen by default
• Introduced timeout for msmtp because without it it hangs completely

Version 2.0.6

• Implemented native capability to push data into InfluxDB in batches. Previous we executed single write() operation for all calls
• Fix Grafana configuration to work with nginx + http auth
• Introduced special option for install tool to install Grafana, InfluxDB and Nginx
• Switched to Stable version of Nginx

Version 2.0.5

• Do not bother with ‘cache folder does not exists. Disabled Netflow cache option’ customers without Netflow enabled
• Introduced new API method show license_hardware_data for getting information about hardware
• Set average recalculation time to 5 seconds for networks too
• Installer: Do not ask user about SPAN if we specified command line argument

Version 2.0.4

• Implemented ability to pass information about pusblished flow spec mitigations with web hook
• Reduced delay for attack detection from 3 seconds to 1 second
• Remove information about subnet’s traffic from email notifications also. It’s useless technical information
• Fix zero UUIDs for attacks in mongodb and for web hooks. Added complete attack description into json web hook
• Use only smoothed (average) network counters for API / Graphite / InfluxDB export
• Introduced new system counters to track number of hosts with traffic and without traffic.
• Added new method for fcli to add new fields into schema for main configuration: upgrade_configuration
• Reduced number of attempts to connect to MongoDB. Removed per-network stats from redis/mongo attack detail
• Implemented ability to run FastNetMon without root permissions. Only for testing purposes.

Version 2.0.3

• Fixed Netflow 9 sampling processing for case when we have non zero scope field
• Fixed bug when we have multiple samplers in one packet. Fixes weird sampling rate for ASR devices

Version 2.0.2

• Added timestamp in RFC 2822 to emails
• Introduced configuration option to tune Netflow cache. Add explicit check that we should have cache folder for Netflow caches
• Add handler for negative padding
• Add support for sFlow packets with padding. We disovered this issue in Brocade ICX6610
• Fix bug in installer. If we have /etc and /tmp on different block devices installer will fail to create configuration files