Although major hacks generate news headlines, most companies and institutions quietly contend ongoing web-based probes and attacks, with the average CISO at the largest businesses managing ecosystems of 50-75 vendors in hopes of catching security breaches.
Level 3, a leading telecom and Internet services provider (also one of the world’s dozen-plus tier one, or backbone networks), has innovated a different solution, one that uses predictive behavior mapping to help stop attacks before they can happen.
“If the Internet is the highway, most ISPs build the roads and on/off ramps, but they aren’t accountable for what cars and trucks do on it,” said Dale Drew, Level 3’s Chief Security Officer. “We decided to take more responsibility for who was driving recklessly on our highway.”
“Then, we could use our network data to find bad guys even before they harm our network or customers.”
It was a radical concept, not only because it challenged the traditional role of ISPs, but also the status quo in the cyber security industry.
“About six years ago, we first got the team together and decided to start looking for patterns in traffic metadata, like IP addresses that had been compromised, and considered doing something about it,” Drew explained. “When we asked security firms to partner with us, they usually demanded exclusive access to the data, and wanted a six month window in which we wouldn’t act, so they could sell an exclusive fix.”
“After the 15th call, on which we got the same basic answer, I had the leadership team around the table, and said ‘we’re now not just in the big data analysis business, but we’re in the security business.’”
Going at it alone would consist of two functions: Detection and categorization of threat risks on the backbone, such as the 10% or so of traffic that evidence the deterministic behavior like skimming IP addresses, and then blocking the IP addresses while notifying the owners of compromised machines.
The ability to detect such “bad guys” required an entirely new infrastructure, new systems, talent, even new code in languages the company had never used before. Stopping them, and notifying potential victims, could utilize existing tech, but necessitated entirely new processes, especially since it would involve notifying customers of competing ISPs, not just its own, but blocking their access to Level 3’s backend until the behaviors were fixed.
“Legal’s concern was that there was no wholly reliable way to distinguish between good and bad traffic [i.e. which “vehicles” to flag], and the team’s biggest fear was we’d mistakenly block some huge DNS server and choke the infrastructure of the Internet,” Drew remembered.
“It helped that our CEO at the time was supportive and gave us permission to lean in, our current CEO, Jeff Storey, continues to support these efforts, and I made a point of keeping our legal team involved.”
A new group was created — Situational Awareness, though now labeled Threat Intelligence — and spent the next 10 months in development and production, working in tandem with the operations team that had experience doing things like IP takedowns. The first test was manual, as company leadership sat around a tech’s computer and watched him hit the enter button to notify a user of suspicious behavior, and shut it down until it was resolved. Automating those functions a few months later “was even more harrowing,” according to Drew.
“We had some interesting conversations early on, like whether or not the notification emails would look like spam. But the act of blocking your IP on one of the world’s largest backbones had significant value.”
“Once a user can’t reach anybody online, they know it’s serious and realize they have to repair their system before online access is restored. This is the cornerstone to ensuring we can convince system owners to properly secure their systems when connecting to the Internet.”
The predictive approach is compelling, since cybersecurity based at a company level is inherently reactive, in that it relies on preparing for compromises (using tactics such as “honeypots” to lure and then segregate them), and is only as good as the knowledge and capabilities of the best services vendor in those sometimes complex infrastructures.
Conversely, Level 3’s systems shut down suspicious behaviors that may not have yet appeared on a company’s radar. It was able to protect all of its customers from the NTP DDOS attack within hours of identifying it in 2014.
The company embeds a portion of its cybersecurity functions in its standard services offering, and recently debuted a cloud-based solution to notify clients of emergent threats, block them on the backbone, and iterate its validation and ranking of threats going forward (called Enterprise Security Gateway).
“We see it as a game changer,” Drew said.