12.04.2020

FastNetMon and Amazon VPC flow logs

FastNetMon was built with flexibility in mind and we do offer security solutions for on premise and cloud environments. Amazon AWS offers great way to export traffic telemetry from all VPC’s (VPC Flow logs) and you can use your FastNetMon instance to process this data easily. Many features from this article (VPC Flow Log, CloudWatch, Lambda) are subject of charge, please check with your financial team before moving forward to avoid unexpected costs.

From FastNetmon side, please enable Tera Flow plugin using following commands:

sudo fcli set main tera_flow enable
sudo fcli set main tera_flow_host 0.0.0.0
sudo fcli set main tera_flow_ports 8104
sudo fcli commit

With such configuration FastNetMon will listen for Tera Flow UDP messages on port 8104.

If you’ve deployed FastNetMon in Amazon AWS too then you need to change security policy for your EC2 instance to allow UDP traffic over port 8104. In instance’s description select “Security groups” and click on it. In this case you need to click on “launch-wizard-2”.

Then you will see all security groups:

Then click on security group to open configuration:

As next step you’ll need to click on “Edit inbound rules” then click “Add rule” and create following rule: “Custom UDP”, “Port Range”: 8104, “Source”: “anywhere”, “Description”: “Allow FastNetMon Tera Flow from Lambda” and finally click on save rules.

Next thing you need to configure is CloudWatch. We need CloudWatch to export Flow Logs into it for processing. Open Logs, Log groups and then from “Actions” select “Create log group”. Use name fastnetmon_flow_logs.

As next step you will need to create IAM Role for Flow logs, please follow official guide for it. When asked for Role Name, please use “vpc_flow_watch_role”.

Then you need to enable VPC Flow logs export for VPCs in all your regions. Open required VPC and then select tab “Flow logs” and then click on “Create Flow log”.

Then set following options for it:

  • Filter: all
  • Maximum aggregation interval: 1m
  • Destination: Send to CloudWatch Logs
  • Destination log group: fastnetmon_flow_logs
  • Role: vpc_flow_watch_role

We may have some rest and check that we’re receiving data for this log group. Let’s open CloudWatch then “Log Groups” and then select fastnetmon_flow_logs. You will see flow data in following format if you’ve configured everything right.

Next step is Amazon Lambda configuration. We need it to run small function which receives every VPC Flow Log, encodes into Tera Flow format and then exports into FastNetMon. Open Lambda setup and click on button “Create Function”, then select “Blueprint” and find “cloudwatch-logs-process-data” and after that click “configure”.

Then please specify following options:

  • Function name: process_vpc_flow_fastnetmon
  • Execution role: Create a new role with basic Lambda permissions
  • Log group: fastnetmon_flow_logs
  • Filter name: fastnetmon_flow_logs_filter
  • Set flag Enable trigger

And after that click on create function.

On next page you will need to set “Runtime” to “Go 1.x”, then set”HandlerInfo” to fastnetmon_flowlogs_lambda”, set option “Code entry type” to “Upload zip file” and upload this file . To specify FastNetMon’s address you need to add Environment variable with name “fastnetmon_server_address” set to “10.10.10.10:8104” (please change to correct external IP address of EC2 instance or another server, please do not use internal IPs from instances here, Lambda does not have permissions to use them). After finishing all steps just click on “Save”.

That’s one more time to check that everything works as expected. Let’s open CloudWatch then select “Logs”, “Log groups” and open “/aws/lambda/process_vpc_flow_fastnetmon”. This log group was created for default logging from our Lambda. We ill use it to confirm that Lambda is working fine. You will see following entries if lambda works fine:

To conform correctness of setup we recommend using following option on FastNetMon to dump every received packet into log file:

sudo fcli  set main dump_all_traffic enable
sudo fcli commit

After that you can monitor all arriving packets with following command:

sudo tail -f /var/log/fastnetmon/fastnetmon.log