Changes

Version 2.0.2

• Added timestamp in RFC 2822 to emails
• Introduced configuration option to tune Netflow cache. Add explicit check that we should have cache folder for Netflow caches
• Add handler for negative padding
• Add support for sFlow packets with padding. We disovered this issue in Brocade ICX6610
• Fix bug in installer. If we have /etc and /tmp on different block devices installer will fail to create configuration files

Version 2.0.3

• Fixed Netflow 9 sampling processing for case when we have non zero scope field
• Fixed bug when we have multiple samplers in one packet. Fixes weird sampling rate for ASR devices

Version 2.0.4

• Implemented ability to pass information about pusblished flow spec mitigations with web hook
• Reduced delay for attack detection from 3 seconds to 1 second
• Remove information about subnet’s traffic from email notifications also. It’s useless technical information
• Fix zero UUIDs for attacks in mongodb and for web hooks. Added complete attack description into json web hook
• Use only smoothed (average) network counters for API / Graphite / InfluxDB export
• Introduced new system counters to track number of hosts with traffic and without traffic.
• Added new method for fcli to add new fields into schema for main configuration: upgrade_configuration
• Reduced number of attempts to connect to MongoDB. Removed per-network stats from redis/mongo attack detail
• Implemented ability to run FastNetMon without root permissions. Only for testing purposes.

Version 2.0.5

• Do not bother with ‘cache folder does not exists. Disabled Netflow cache option’ customers without Netflow enabled
• Introduced new API method show license_hardware_data for getting information about hardware
• Set average recalculation time to 5 seconds for networks too
• Installer: Do not ask user about SPAN if we specified command line argument

Version 2.0.6

• Implemented native capability to push data into InfluxDB in batches. Previous we executed single write() operation for all calls
• Fix Grafana configuration to work with nginx + http auth
• Introduced special option for install tool to install Grafana, InfluxDB and Nginx
• Switched to Stable version of Nginx

Version 2.0.7

• Added api_port configuration variable
• Fixed bug in schema upgrade code inside CLI
• Install pwgen by default
• Introduced timeout for msmtp because without it it hangs completely

Version 2.0.8

• Introduced ability to read MongoDB configuration options from special file in fcli
• Introduced option to specify custom mongodb port for fcli
• Moved configuration for mongodb for fcli to special file

Version 2.0.9

• Added upstart file for tera-tsdb

Version 2.0.10

• Completely working ASN lookup for traffic database
• Introduced src_asn and dst_asn fields for traffic db and simple packet
• Implemented support for pushing data into Clickhouse

Version 2.0.11

• Since this commit we retrieve attack bandwidth information from time when atatck was detected. This commit fixes cases when customer report that attack report was below threshold
• Removed Average packet size for incoming traffic and Average packet size for outgoing traffic from attack details report
• Unified smtp notifications. Do not block thread with email process. Let’s fork additional threads for this task
• Introduced ability to remove packet length if we detected fragmentation attack. Implemented facility for rule optimisation
• Added requirement to run fill_dictionaries from root

Version 2.0.12

• Introduced ip_length field to track only IP packet total length (without Ethernet). We need it because BGP Flow Spec uses such representations

Version 2.0.13

• Fixed issue: netflow_host ignored in previous releases

Version 2.0.14

• Moved message about absence of template to DEBUG level because customer complaints
• Introduced additional log messages to debug Netflow templates delay

Version 2.0.15

• Print ASN mapping file version to log file
• Add special check that we could de-serialize data from ASN mapping table

Version 2.0.16

• Fix bug in traffic presistency. Add correct traffic directions. Previously we had other everywhere

Version 2.0.17

• Introduced ability to push network’s traffic to Clickhouse
• Introduced ability to configure Clickhouse for metrics using configuration interface

Version 2.0.18

• Implemented support for extracing IPv6 total counters using API: show total_traffic_counters_v6
• Introduced special counter for non IPv4/IPv6 traffic: non_ip_packets.
• Introduced option to disable/enable IPv6 traffic processing explicitly: process_ipv6_traffic

Version 2.0.19

• Allow small memory differences for license server
• Implemented network_counters_v6 in fcli
• Implemented ability to get sorted average counters for IPv6 subnets
• Fix bug in IPv6 get_packet_direction_ipv6. Now we could correctly detect outgoing traffic

Version 2.0.20

• Introduced ability to execute notify script when attack mitigated with blackhole (for ban and unban). Also, introduced ability to call it when attack was mitigatied with flow spec (script call on announce withdrawal is not supported yet). Please set notify_script_format to json to use this option
• Introduced variable to configure format for notify script. Now we support text and json formats

Version 2.0.21

• Introduced very nice optimization. Do not write zero elements to Clickhouse and InfluxDB
• We introduced show latest_fastnetmon_version for fcli to get latest version of toolkit from server
• Introduced ability to retrieve FastNetMon running version from fcli: show fastnetmon_version

Version 2.0.22

• Fix bug with vendor specific implementation issues. When we have padding at the end of sFlow packet we drop all frames. Now we process all of them correctly

Version 2.0.23

• Introduced method license_hardware_data_reset to reset hardware assignment. You allowed to do it only three times

Version 2.0.24

• Implemented an ability to specify number of hosts in output of show host_counters with env variable: HOST_COUNTERS_MAX_HOSTS

Version 2.0.25

• Implemented full support for fcli command show sflow_sampling_rates. You could use it for sFlow sampling rate monitoring

Version 2.0.26

• Implemented host_counters_v6 in fcli
• Implemented method to export top IPv6 talkers

Version 2.0.27

• Fixed whitelising option (main)

Version 2.0.28

• Rename process name fastnetmon to be unique line because it conflicts with Monit’s entity created for server itself
• Disabled graphite by default. If somebody wants it he could enable it manually
• Report incorrect value in field gobgp_flow_spec_default_action

Version 2.0.29

• Introduced autconnect logic for traffic persistency backend

Version 2.0.30

• Implemented batch inserter for traffic persistency backend (significantly faster than old implementation)
• Upgraded persistency database library integration. Fixed crash in case of connection issues with database

Version 2.0.31

• Fixed bug with active option for BGP peers. Now we handle it correctly
• Added support to store IPv6 address in tera flow format and inside tarffic persistency backed

Version 2.0.32

• Added new plugin to read data in tera flow format from external sources (another FastNetMon instance).
• Implemented enhanced error handling for write_simple_packet
• Introduced filter to count full number of packets processed by process_packet function: total_simple_packets_processed

Version 2.0.33

• Fixes bug with zero packetDate in traffic db
• In case of issues with connection with our traffic persistency backed FastNetMon could crash
• Hide email password from output of show main command

Version 2.0.34

• Added option flow_spec_tcp_options_use_match_bit (main) to enable match bit in BGP Flow Spec announces. Some vendors require this flags
• Traffic DB got an ability to create required databases/tables for traffic persistency automatically

Version 2.0.35

• Introduced configuration option flow_spec_fragmentation_options_use_match_bit (main). You could use it to enable match field inside BGP Flow Spec announces. Some vendors require this flags
• Reworked code which handles BGP Flow Spec Fragmentation Flags

Version 2.0.36

• Added new configuration option flow_spec_do_not_process_length_field (main) to suppress any processing for length field. It’s useful in case when your device send incorrect IP length in Netflow/IPFIX

Version 2.0.37

• Added new function for fcli to lookup host group for specified IP address: show ip_hostgroup 11.22.33.44
• Upgraded HTTP library to new version. We use it for web_hooks and InfluxDB metrics export

Version 2.0.38

• Improved compatibility with SMTP protocol in email notifications
• Added unique Linux thread names for all Netflow capture threads
• Fixed bug which caused negative source_id in template cache

Version 2.0.39

• Introduced ASN lookup function for fcli: ‘show ip_asn 8.8.8.8’ you can use it to get ASN for any host

Version 2.0.40

• Added option to add simple packet dump to email: email_notifications_add_simple_packet_dump

Version 2.0.41

• Improved license validation code to ignore small differences in hardware configuration

Version 2.0.42

• Added initial support for systemd

Version 2.0.43

• Added fcli to PATH

Version 2.0.44

• Significant performance optimisation for users with big networks. 1.9 second => 0.7 second for /8 network
• Performance improvement: reduced lock contention for DDoS checks
• Added number of host groups to info log
• Performance improvement. 3.5 seconds -> 1.9 seconds for 1m hosts for speed recalculation. Replaced copy by value to copy by reference

Version 2.0.45

• Improvement for users with huge networks lists: we do not add small networks already added into lookup table as part of big networks

Version 2.0.46

• Made BGP Flow Spec valdiation configurable with configuration option: ‘flow_spec_execute_validation’
• Introduced option to enable/disable network aggregation to reduce number of networks: ‘aggregate_networks_list’
• Disabled configuration check for FastNetMon in systemd/Upstart unit

Version 2.0.47

• Introduced separate repo for Ubuntu Trusty Tahr

Version 2.0.48

• Added option to list all interfaces availible in system: show interfaces
• Fixed BGP daemon autostart unit

Version 2.0.49

• Introuced per protocol counters export to InfluxDB: influxdb_per_protocol_counters

Version 2.0.50

• Introduced an ability to skip export of host counters to reduce lod on InfluxDB: influxdb_skip_host_counters
• Added complete ignore to SIGHUP signal
• Added suport for Ubuntu 16.04 for graphic stack

Version 2.0.51

• Added host group name to email/json notification about attack

Version 2.0.52

• Added options dump_all_traffic and dump_other_traffic to dump all/other traffic to log for debugging reasons
• Introduced new option sflow_use_new_generation_parser which could enable new packet parser for parsing sFlow packets

Version 2.0.53

• Improved license validation code on grey IPs

Version 2.0.54

• Introduced support for md5 secured BGP sessions: md5_auth, md5_auth_password

Version 2.0.55

• Added debug messages about orphaned buckets

Version 2.0.56

• Enabled batch mode for Clickhouse by default
• Added automated deployment for all dashboards for Grafana
• Implemened an ability to create InfluxDB data source with API in Grafana automatically
• Improved InfluxDB installer to create database with proper retention

Version 2.0.57

• Introduced JSON mode for fcli: JSON_MODE=on sudo -E ./fcli show bgp
• Added JSON methods for almost all fcli commands

Version 2.0.58

• Enabled code dump handlers for fill_dictionaries for debugging purposes

Version 2.0.59

• Enabled API gateway by default

Version 2.0.60

• Fixed segmentation fault for fill_dictionaries: double memory free up

Version 2.0.61

• Added automatic schema cration for Clickhouse

Version 2.0.62

• Introduced traffic_db_sampling_rate to configure sampling rate for AF_PACKET capture

Version 2.0.63

• Enabled IPv6 by default in traffic_db
• Fixed bug with traffic_db. It crashed when we did not have Clickhouse running when traffic_db starts
• Added check for correctness of networks in cidr for for fcli

Version 2.0.64

• Implemented option to export host counters using API

Version 2.0.65

• Complete web API
• Automatic creation for all Clickhouse tables
• Introduced ability to create database for Clickhouse metrics automatically
• Added correct check to handle unreachable Clickhouse for metrics export

Version 2.0.66

• More robust license check code

Version 2.0.67

• Introduced complete support for licenses on grey IP addresses

Version 2.0.68

• Implemented support for multiple interfaces in AF_PACKET
• Added new configuration option af_packet_read_packet_length_from_ip_header to read size from IP header instead of wire
• Added option for strict CPU affinity for AF_PACKET: afpacket_strict_cpu_affinity

Version 2.0.69

• Re-introduced fastnetmon_client toolkit to emulate old tool from FastNetMon Community
• Added symlink for fastnetmon_client to call without full path
• Introduced fake mode for fastnetmon_client using key z for example output

Version 2.0.70

• Netflow plugin: implemented ability to read src and dst ASNs directly from Netflow v9 packet
• Traffic db will not try to fill ASN information if we already have non zero ASN from FastNetMon

Version 2.0.71

• Fixed bug with ASN population code

Version 2.0.72

• Implemented ability for Netflow v9 to read input and output interfaces for packet from flow data
• Netflow v5. Added ability to read input and output port numbers from Netflow packets
• Added two new fields for Clickhouse schema: inputInterface and outputInterface

Version 2.0.73

• Introduces IPv6 mode in fastnetmon_client -ipv6

Version 2.0.74

• Migrated to native Go interface code which creates fastnetmon user for MongoDB
• Installer does not remove old configiration. It just renames it
• Added autostart for BGP daemon
• Complete support for IPv6 for notify json script and web hook
• Implemented ability to show banned IPv6 address

Version 2.0.75

• Full IPv6 support
• Fixed whitelisting for IPv6
• Fixed segmentation faults when fastnetmon load IPv6 network into whitelist

Version 2.0.76

• Added basic ssh server into fcli: sudo -E SSH_SERVER_MODE=on ./fcli

Version 2.0.77

• Added community support for IPv6 mode
• Added gobgp_community_host_ipv6 and gobgp_community_subnet_ipv6 to configure IPv6 independently
• Working IPv6 BGP integration
• Added configuration options for IPv6: gobgp_announce_host_ipv6 and gobgp_announce_whole_subnet_ipv6 to configure behaviour independently to IPv4
• Added field gobgp_next_hop_ipv6 to configure IPv6 next hop
• Implemented option to create IPv6 blackhole using cli
• Added IPv6 unicast and flow spec AFI for GoBGP.

Version 2.0.78

• Added network for host into JSON and text/email notifictions
• Added protocol version to JSON and email/text notify scripts
• FastNetMon can populate host group properly for manually created blackholes
• Migrated MongoDB version 3.6

Version 2.0.79

• Implemented API and fcli to retrieve traffic for single host: fcli show single_host_counters 192.168.1.100

Version 2.0.80

• Enabled option to use IPv6 only peers

Version 2.0.81

• Added ability to override Router ID for BGP peers: gobgp_router_id Mandatory for IPv6 only setup.
• Explicitly added local_address in BGP configuration

Version 2.0.82

• Added support for accept BGP Flow Spec action type
• Unified upstart/systemd configuration for new Ubuntu distributions

Version 2.0.83

• Improved compatibility with Ubuntu 18.04

Version 2.0.84

• Added ability to change traffic_db configuration using configuration file
• Added ARM64 compatibility
• luajit code upgarde for ARM64
• Added ARM64 version of installer tool
• Switched BGP daemon logging to stdout for systemd enabled distros

Version 2.0.85

• Added code for BPF sampling in AF_PACKET filter: mirror_af_packet_sampling_rate

Version 2.0.86

• Migrated to per day partitions for traffic_db/Clickhouse
• Removed dependencies on libpcap and libnuma from deb package

Version 2.0.87

• Added per day partitioning for Clickhouse metrics

Version 2.0.88

• Added configuration option flow_spec_do_not_process_source_address_field. It’s very useful to provide good level of aggregation for memcached/ssdp attack types

Version 2.0.89

• Added support for flow_spec_do_not_process_length_field and flow_spec_do_not_process_source_address_field for flow spec mitigations in sFlow/AF_PACKET modes

Version 2.0.90

• Introduced configuration option to configure logging level: sudo fcli set main logging_level debug
• Fixed parser for flow spec detection enging for fragmented packets

Version 2.0.91

• Moved logging level reconfiguration close to toolkit run

Version 2.0.92

• Boost library upgrade to fix locale related issues and add new functions

Version 2.0.93

• Added https support for web hook handlers

Version 2.0.94

• Added IPv6 support for Netflow v9 code

Version 2.0.95

• Added export for system counters to InfluxDB: set main influxdb_export_system_counters
• Added options for dpkg to suppress interactive reconfiguration attempts

Version 2.0.96

• Added variables netflow9_custom_sampling_rate_received and netflow9_sampling_rate_changes to track sampling rate learning for Netflow v9 better
• Added netflow9_options_packet_number counter to debug Netflow issues
• Initial support for viewing all availible options of fields in api

Version 2.0.97

• Added support for duration histogram for Netflow v9

Version 2.0.98

• Added ability to configure speed recalculation delay with option speed_calculation_delay. It’s quite useful for debugging and for customers with really huge networks

Version 2.0.99

• Added support for input and output interfaces for IPFIX
• Added src and dst ASN support for IPFIX
• Improved error reporting for Graphic stack installer
• Added support for Clickhouse server automatic install with: -install_traffic_persistency

Version 2.0.100

• Added agent ip address for Netflow v5
• Added agent ip address for Netflow v9 and IPFIX
• Added new field AgentIP address for Clickhouse
• Enabled support for 18.04 in installer and switched 18.04 to MongoDB 4.0

Version 2.0.101

• Added default fields for metricDate for Clickhouse metrics to avoid timezone issues
• Added automatic calculation for packetDate in traffic_db. We did it to eliminate timezone issues
• Added https support for apt in installer
• Enabled Ubuntu 18.04 by default
• Added support for 18.04 for graphic stack installer

Version 2.0.102

• Reintroduced Redis support

Version 2.0.103

• Added fcli/API command: bgp_incoming_announces to expose received BGP announces from peer

Version 2.0.104

• Fixed BGP Flow Spec validation bug for rules injected from API/fcli

Version 2.0.104

• Added IPv6 support for IPFIX

Version 2.0.106

• Implemented sudo fcli show netflow_sampling_rates to expose sampling rate information for Netflow v9 agents
• Added support for duration distribution for Netflow v5

Version 2.0.107

• Added option to switch connection tracking from flow to unique oppiste hosts tracking with sudo fcli set main connection_tracking_skip_ports

Version 2.0.108

• Upgraded Clickhouse library to new version (fixes retries and timeouts)

Version 2.0.109

• Added ability to disable ban actions for incoming/outgoing traffic using: do_not_ban_outgoing and do_not_ban_incoming

Version 2.0.110

• Fixed bug for Netflow v9 processing code which caused infinite loop after receiving malformed packet

Version 2.0.111

• Added command to configure mongodb initial configuration from fcli using fcli create_configuration