An in-depth discussion on all things DDoS
In this episode of the Router Networking Podcast, James Dean interviews Pavel Odintsov, co-founder of FastNetMon. Pavel shares his journey from a curious child dismantling audio equipment to becoming a key figure in cybersecurity. He discusses the challenges of dealing with DDoS attacks, the evolution of network security, and the importance of understanding fundamental technologies. Pavel also highlights FastNetMon’s role in identifying and mitigating DDoS attacks, offering insights into the future of network security. Tune in to learn about Pavel’s experiences and his advice for aspiring tech professionals.
Listen to the podcast on our YouTube channel
Read the podcast transcript:
James Dean: Hi, my name is James Dean. You’re listening to the Router Networking Podcast. Today I’ve got a special guest, Pavel Odintsov, who is the co-founder of FastNetMon. Today we’re going to learn all about Pavel’s background, how he got started in the security space, what he’s up to now, and where he sees the future of security going. How are you, Pavel?
Pavel Odintsov: Hello, hello. I’m just wonderful. Thank you for inviting me. I’m happy to talk about my story, about FastNetMon in general, and everything happening in the cybersecurity field.
James Dean: Perfect. Yeah, it’s been good to finally speak. We’ve had the Corbett team for a couple of months now, right? So it’s good to get this set up. One place I always like to start, Pavel, is how did you get started in the world of security?
Pavel Odintsov: Oh, actually, it was a long story for me. I got a formal education in computer science and then started working as expected. I got an internship, then my first job as a Perl developer. It was focused on software development. I had requirements, a list of tasks, and many weeks to implement what I was asked to do. It was old-style software development, but by accident, my first company was a domain name registrar, and my first project was to implement billing integration with a new department introduced by the company. It was a hosting department. That’s how I was dragged into not just software engineering. After that, I spent around five years in heavy operation of Linux operating systems. I started as a software engineer and moved to operations. Then I was invited to join a company in the cloud compute area as a technical co-founder, as CTO, and it was the first place where I met cybersecurity. Our company operated in very low-margin hosting services. It had plenty of customers, about 25,000 in total, but very cheap, like on the minimum possible borderline of what you can afford to sell. Can you imagine we had lots of problems with those? At first, it started like one attack per month. It was something incredible. Everyone thought, wow, it’s exciting. I mean, it’s dangerous. It’s a broken network. After a few more months, we accepted that once per month, we have some issues with the network. In the coming months, it became once per week, and at some point, when I left this company, it was around six attacks on a daily basis every single workday. This is how I was dragged into cybersecurity. At first, it was a minor issue for me as CTO, as a system administrator, but as the days and months went by, it became more important. At some point, I was involved. I wouldn’t say it’s full-time, but maybe 20% of my time was allocated just to deal with cybersecurity issues, to talk with network operators, to find out what we can do and how to stop them. So I would say I was dragged into cybersecurity by the urgent need to deal with problems.
James Dean: Yeah, it’s funny, a lot of people don’t set out to do the role they’re now in, but it just seems to happen when they worked at a company, and that came about. So yeah, obviously, you were pursuing a career as a software engineer in the first place. What drew you to the software engineering space?
Pavel Odintsov: It wasn’t a hard choice because, since childhood, I started what I would call the start of my career. I started at home by disassembling my father’s sound amplifiers and different audio systems. I was curious about what’s inside. It’s one of the things that follows me, like I need to check what’s inside, how it works, what kind of magic happens inside. But the problem was I was just wonderful at disassembling stuff. Sometimes my father was able to put it back. But at the final steps of my career, I was about eight years old, and my father just looked at what I did and gave up. He couldn’t put it back because it was disassembled so hard. So, you know, with my serious skills in disassembling stuff, at some point, you need to do something on your own, assemble something. I had problems with this kind of option. So, when I finished school, I thought maybe I need to learn how to actually do something good instead of just assembling stuff. That’s how I decided to join a computer science course at university.
James Dean: And when you first started university with computer science, apart from breaking things and putting them back together, what made you passionate about that?
Pavel Odintsov: Back in time, I had a major passion for physics.
James Dean: Yeah.
Pavel Odintsov: And that’s how I started my education at university. Not directly, not from the first time as a computer science student, because I spent around one course as an applied physicist. It was a great experience because, in school, I was passionate about physics and all that stuff. Then maybe it was a great idea because my local university introduced a very new shiny course about physics. Brand new, not typical stuff because universities usually have quite old courses, old programs. But this one was shiny, everything was new. They had a course on Linux, can you imagine? That was 2005. Linux, C language, lots of distributed systems. It looked exciting. I had a great deal of interest in how I could assemble stuff. Applied physics is quite close to what you assemble, like literally everything. But quite soon, I realized that it’s not actually my cup of tea. Then I got confirmation that in school, I had a passion for physics, but it had nothing to do with applied physics. So I switched to computer science.
James Dean: You started a business very young. I know you touched on it earlier, but you co-founded a business pretty much out of university, is that right?
Pavel Odintsov: Yes, absolutely right.
James Dean: What was the motivation to do that then? Most people would just get into a grad scheme or…
Pavel Odintsov: Oh, it was a weird story. I started working in my third year of university. It wasn’t a part-time job; it was a full-time job. I was born in a small town called Samara, literally in the middle of nowhere. The problem is we had no great selection of shiny computer companies. Back in my university times, it was about selling computers, not writing software. It was amazing because most of the stuff about computers around me was like, we can do a website for you. That’s all, the first offering of technological companies in my hometown. But at some point, my friend, actually my classmate from the physics course I started, we kept in touch. He said, we have a great company you may be interested in because they have Linux and Perl, but their position was full-time only. It was like a chance, one in a million. In my hometown, it was literally impossible to find any kind of software engineering job. But this one was great, great technology, great stuff. I had lots of problems with my university. I was about to be kicked out about three times, but I survived it.
James Dean: Oh, because your grades were so bad.
Pavel Odintsov: You know, when you try to work full-time at university and a full-time job, you need to sacrifice something sometimes. But I wasn’t able to sacrifice my job because I didn’t expect that I could get such a great job. It was like winning millions of pounds. Just keep it, stick with it.
James Dean: So you finished uni and then started your own business pretty much.
Pavel Odintsov: Straight away. I joined as a co-founder after having two years at this job. I worked at this company, and they offered maybe we can work together in the cloud compute area. Back in time, it was just a dream job, but now they think of me working on my own stuff again. I just went for it. It’s exciting.
James Dean: It’s quite exciting. I imagine at such a young age as well. How big did the company get to?
Pavel Odintsov: So far, this company still exists. They have about 45 people, but when I left, we had 25 folks. Most of them were in technical support because it’s a hosting company, and the main area of what you need to do is answer customer questions. Many of them had issues with Apache, PHP, MySQL.
James Dean: You mentioned earlier about the attacks going from once a month to six a day. Attacks have been an ever-growing part of the security space. For you personally, being involved at such a young age and being a co-founder of a business at such an age, what do you think has been the most significant changes in networks or in security since you’ve been involved?
Pavel Odintsov: So I would say, yeah, for the first time, when at my company we started getting the DoS attacks, I started talking with friends in my industry, like from telecoms, data centers, and many of them were like, no, don’t see them that often. So are you talking real stuff, or are you just exaggerating, like so many attacks per day, daily basis? It just cannot be because even for large folks from large companies, you don’t see them. And only one company who actually trusted my complaints and my questions, it was my ISP, and they saw exactly the same number of attacks. You can see like you are definitely not exaggerating. You have many of them, and many reasons why we had it this way because I suppose we had quite a large and diverse audience of our customers from many countries, from many backgrounds, many of them were from the gaming industry. It was just emerging. There were many Minecraft servers, Counter-Strike servers, and they’re very well known to attract so much DDoS attack. But what changed now, everyone perfectly knows about DDoS attacks. Like all people in technical industries, they experienced them firsthand. And now it’s ever more different because sometimes we also can hear about businesses completely devastated and ruined by DDoS attacks.
James Dean: I’m not massively technical myself, so it’d be good for you to give me an insight. But also, some of the listeners on here may not know specifically what a DDoS attack is. So, what is a DDoS attack?
Pavel Odintsov: So, a DDoS attack, to be precise, is distributed. The terminology is quite misleading sometimes because typically people use DoS and DDoS. But it’s a problem because it doesn’t explain everything you’re working with. Because distributed, it involves at least thousands of different devices attacking your website or your service. It’s typically a small number of requests, but because of the number of them, and typically like theories, they’re attacking for attacking the it’s not prepared to this number of requests. For example, it may be like, for example, it’s just maybe the site of a local council, and they have maybe 20,000 residents. But if some bad guy starts an attack having like 100,000 fake residents just opening their main page or just opening how I can move into your borough or what’s happening. And because of scale, because of the number of them, it just knocks the service down. And so, first of all, it is to exponential spike of resource consumption for some short time. Services may be able to handle because all the time they have some spare capacity because, you know, every single service is built to have some gap, some spare capacity to make it a little bit more like we have on average maybe 100 visitors for a day. But typically, services handle like maybe a thousand visitors, but not 10,000 or 100,000. And at some point, it just knocks the service down. And that’s all. And at this point, you need to do some magic. You need to allocate more resources. You need to hire maybe some special company or try to do magic or deploy FastNetMon to understand because one of the great problems is DDoS attacks. So I would say it started happening quite recently, maybe a few years ago. People just blame DDoS attacks for their faults like misconfiguration. It’s called DDoS.
James Dean: Now, why would someone do a DDoS attack though? Is it just out of vendetta, or is it to try and dip into other parts of the site?
Pavel Odintsov: I would call it a gray area. My understanding is limited because DDoS attacks are very distributed, coming from different countries and services. It’s extremely complicated to find out who actually started it. If someone buys a DDoS attack from bad actors, the person you bought it from will never start the attack themselves. They’ll go to another person and buy it again. There may be tens of people working on different levels, and only at the end of that line is the person who actually starts the attack. Most of the time, it’s hate. I don’t like this site, what they are talking about, their idea, or this specific person. It’s quite common with gamer sites. If your clan failed in some competition, it’s one of the things.
James Dean: Yeah, yeah.
Pavel Odintsov: In developing markets, it’s competition. It typically happens in extremely competitive and fast-growing markets. For many developing countries, the Internet is just emerging. Some densely populated areas just got Internet, and people see it as a gold mine. They think they can sell Internet and earn thousands of dollars. Then they start thinking, maybe we can knock them down and get more customers.
James Dean: Oh yeah, it’s one of those things. If you’re not involved in it daily, you don’t really hear about it unless it’s a really bad attack. So, you mentioned the big changes in the DDoS space since you’ve been involved. What’s most concerning for you right now?
Pavel Odintsov: One of the most concerning things is that DDoS attacks are typically run by a protocol called UDP. The problem is that the IETF recently introduced a new version of the HTTP protocol, HTTP/3, which uses UDP. Back in the day, it was simple to filter out DDoS attacks by filtering out UDP. UDP isn’t widely used outside of networks. You can run your own DNS service or have a local VoIP service, and then you can ban everything else and be safe. But because HTTP uses UDP, we can’t filter it out anymore. It eliminates the option for simple and cheap DDoS filtering for small companies. They need to move to more sophisticated and expensive equipment to filter it out. We need to improve tools and find out what we can do. Many companies have suffered from it and can’t use their old approaches for DDoS mitigation.
James Dean: That’s one of the things you’re working on at the moment with FastNetMon. What does FastNetMon do?
Pavel Odintsov: FastNetMon solves one of the first problems with DDoS attacks. When you experience DDoS attacks, your network is often dead. If you work from home, you can’t access your network to check what’s happening. You need to find out what is the target of the attack. Even small networks have tens of thousands of different hosts, typically with different IPs and services. FastNetMon helps you find out which IP is under attack. Instead of logging into your equipment and running TCP dump, FastNetMon will tell you which IP is under attack. It can also automatically mitigate the attack. FastNetMon can create BGP announcements to do different things, depending on what the company wants to do. It automates actions. We receive an attack, and FastNetMon can create a BGP announcement with specific community, IP, and next hop. It’s up to the network administrator what they can do. Sometimes they want to blackhole it, literally stop all traffic to this host. It may sound harsh, but it saves the network. In a calmer environment, we can think about what to do next. Using BGP, specific services can be moved to a scrubbing center. A scrubbing center has equipment for filtering DDoS attacks. FastNetMon can create BGP panels to move malicious traffic away from the normal path to equipment prepared for DDoS attacks. The last resort option is using capabilities from routers. It’s called BGP FlowSpec. Most modern routers like Juniper MX series, Cisco SR series, Nokia series, and Huawei support BGP FlowSpec. It’s a distributed way to make changes for ACLs using BGP protocol. FastNetMon can help with it. For example, if your network is under attack from DNS amplification, which typically involves fragmented UDP packets, FastNetMon can find out the problem and create a specially crafted BGP FlowSpec to filter out malicious traffic.
James Dean: So you would have customers come to you pre-DDoS attack. You don’t get called in if something really bad happens, it’s too late by then.
Pavel Odintsov: People never solve a problem if they don’t have one.
James Dean: Yeah, true.
Pavel Odintsov: We have only a small fraction of customers.
James Dean: What’s next for the business? Where do you see FastNetMon going next?
Pavel Odintsov: We have a great deal of plans. We focus on two areas. First, of course, is DDoS. We’re passionate about DDoS. We work hard to add more support for different vendors and equipment, make it better, and make deployment faster. The second area is network visibility. To detect DDoS attacks, we need lots of metrics. We realized we have so many metrics. We have counters about per protocol distribution, per network, per host, per group of hosts. We have top talkers, peering reports, graphs about peak usage for specific interfaces, routers, and upstreams. It’s a great addition for DDoS mitigation because after we mitigate an attack, we need to understand what happened, why it happened, and what the impact was. Traffic visibility is crucial to have in place after you mitigate a DDoS attack because it provides lots of information about what happened during the attack. DDoS attacks have become more sophisticated. Sometimes they can do a DDoS attack just to move attention away from specific servers. It happened with Band. Bad guys started a DDoS attack, and everyone started running around, but at the same time, bad guys stole all their secrets. Accounts and transactions were very quiet and relaxed because nobody cared.
James Dean: What’s been the most challenging disaster you’ve personally encountered in your career?
Pavel Odintsov: It wasn’t technical. I remember the first company I founded, all network engineers left without any notice. I wasn’t a network engineer. I was CTO, so I was in charge of DDoS, but I wasn’t the guy who actually handled BGP routers and all that stuff. But next month, I learned how to do it because it was the only option. We started hiring new people, but it’s not easy to find a network engineer. We worked in a not very large town, so we had no great choice. We started interviewing people, but at the same time, we needed to keep the network running. It was like emergency onboarding into serious networking. By the end of it, I was in perfect condition, knowing how Quagga works, what kind of BGP communities I need to set, and how to create VLANs. It was a great experience, but I would say it was the most stressful month of my life.
James Dean: Just become a network engineer in a month. Yeah, I bet that was hard. Not just becoming a network engineer in a month, but with your whole career, how do you think your computer science degree played a part in that?
Pavel Odintsov: It was very helpful. I started working in my third year of university, but the first and second courses provided a great background. It was graph theory, computational theory, algorithms, lots of basic stuff you must know. Many people may say they’ll never use compute theory, but I found out during my career that when you work in deep tech areas, you can’t just Google stuff. If you’re a popular technologist, you may think you don’t need to know how computers work, but I find the same approaches still apply even after 10 years. Microprocessors became faster, but they didn’t change. If I need to understand why something is slow, I can use my knowledge. It was very useful. In Britain, it’s fine to get basic knowledge for two or three years and then keep your hands dirty. That’s what happened with me. I got a job, and it was a great choice.
James Dean: We’ve had previous guests say you can learn as much as you want in a book, but unless you do it, you won’t pick it up. You’ve expressed that, learning full-time at uni while working a full-time job. It’s probably the best way to pick up what you’re learning at uni. Is there any particular advice you’d give someone now who’s an 18-year-old looking to get started in the industry?
Pavel Odintsov: We have a choice. We can focus on shiny products, shiny companies, shiny technologies. During my career, I’ve witnessed many sunrises and sunsets of companies. My career is not that long, around a decade. I’ve witnessed the death of many technologies that were the best possible when I was at university. If you look around and see a shiny technology, it will likely be dead in 10 years. My recommendation is to focus on stuff that will not change. Learn how networks work inside, how BGP is implemented, what kind of internals, read software, read code, understand how operating systems work. This knowledge will last way longer than any vendor or product. My best example is the protocol called DNS. It recently crossed 36 years, and it still works. If you decided to be a DNS engineer 36 years ago, you could make some money, not get rich, but some money.
James Dean: It’s funny you’d say that because some of the guys we’ve had on the network engineering side, their advice is not to focus on the shiny objects but actually understand the basics of networking, how a packet goes across a network, and the basic routing protocols. It’s funny you’d say that for your specialty as well. Is there anything you would advise your younger self?
Pavel Odintsov: Keep doing what you’re doing. I made lots of mistakes, but I learned from them. Mistakes are one of the best ways to learn. You don’t learn from success stories. We need to mess up something seriously.
James Dean: You’ll make some serious mistakes. No, you’re right. I appreciate that. It’s been great speaking to you today. We’ve got a quick-fire round to go. This part is five questions, just a bit about you personally. It’s called Quick 5, but you can go into as much detail as you wish or be as quick as you want. If you could describe yourself using just three adjectives, what would they be?
Pavel Odintsov: Loud, dedicated, persistent.
James Dean: Perfect. How much caffeine would you say you consume in a day?
Pavel Odintsov: On average, one cup. If I do two cups, my day will be ruined because I’ll be shaking, and I need to do something, but I can’t because I’m shaking.
James Dean: You get too energetic for me.
Pavel Odintsov: Yes, exactly. I’m already crazy, and I don’t need coffee to make it worse.
James Dean: Apple or Android products?
Pavel Odintsov: iPhone. Many years ago, I moved away from Mac OS and MacBooks in general. They’re getting better now, but it was too complicated for me as a software engineer. I need to develop on hardware and need access to hardware. That’s why I work on Linux. From iPhone, it’s just convenience. It works fast, works well, and can survive a few years of my heavy use. I look for things to be simple and not need much attention. If it starts asking for too much attention or breaks too frequently, I’ll look for different options.
James Dean: Do you prefer working from home or in the office?
Pavel Odintsov: Something in between. Since the start of my career, I’ve worked remotely because it’s the only way I can keep my university place and work for a company. But I miss people. I want to discuss stuff, make planning meetings, exchange ideas, and test ideas. I hate working from home.
James Dean: I hated it. Anyway, I don’t mind at the weekends if I have a bit to do. I’ll just put my headphones on and crack on for a bit, but generally speaking, I do much prefer the office. What was your favorite subject in school?
Pavel Odintsov: Physics.
James Dean: And you’ve carried it on in some way. Perfect. Pavel, it’s been great getting to know you better, hearing more about your career. I didn’t know too much about DDoS, being more of an IP recruiter myself. I’ve learned a lot today, and I hope the listeners have too. If anybody wanted to reach out or had any questions for you, where’s the best place they can find you?
Pavel Odintsov: Just Google my name, and you’ll find lots of options. LinkedIn, I have open messages, message me anytime. I’m quite active on Twitter, and of course, FastNetMon.com is the best option to reach me, to ask questions. Because we’re an open-source project, my account on GitHub is Pavel_Odintsov. We have around 10 different social media accounts, rooms, chats. Just Google them, join any of them, and you’ll meet many nice people, including me. Sometimes I can answer stuff, but more often, someone from the community will provide better, faster, and more skilled answers than me. That’s the power of communities, that’s how it should work.
James Dean: I really appreciate your time, thank you very much, and have a great weekend.
Pavel Odintsov: Thank you so much, you too.
James Dean: Bye.