The Story of our KAI Artificial Intelligence Ad Fraud Prevention

August 06, 2020


John Wanamaker, a retail tycoon during the 19th and 20th centuries is credited with saying,

“Half the money I spend on advertising is wasted; the trouble is I don’t know which half”

Nearly 100 years since his death the advertising industry is still asking itself the same question.

Although digital advertising was originally framed as an exception to this statement, it merely created new paths for wastes of advertising dollars to occur.

The biggest one is what the industry terms “Ad Fraud”.

What is it, exactly?

[Ad Fraud – fraudulently representing online advertising impressions, clicks, conversion or data events in order to generate revenue.]

Last year alone it was estimated that Ad Fraud could account for losses between $19 and $42 billion, with some industry insiders calling those figures conservative at best.

So severe is the problem that an entire sub-industry developed just to address it called Ad Fraud Prevention, yet despite the numerous companies and software therein, Ad Fraud losses continue to rise.

In 2016, Kubient threw its hat into the Ad Fraud Prevention ring. 

The Kubient team began by seeking to solve problems around latency, which was causing video ads to fail due to prolonged load times, and crashing our partner’s web site/app properties operating on third party technology we were licensing. The latency load time spinning wheel of death was not only preventing video ads to render properly, but dramatically affected the user experience and ad inventory yield.

We made the decision to design and build our own technology to solve these issues. Building your own advertising technology stack is not for the faint of heart—systems need to be compatible with industry standards relating to protocols and speed.

300 milliseconds (ms) is the benchmark to start and end the sale of a single ad unit or auction, such as in a banner or video ad.

When we released the first public version of our software with Yahoo as the first advertising partner in June of 2016, it was accomplishing the auctions in 100ms or less, leaving a remarkable 200ms to spare.

With this big of a speed difference between the industry standard in loading times, and what Kubient was able to do from the first go, it was clear this platform was destined for greater things related to the latency issues that plagued the adtech industry. So we started asking our partners what the top 3 issues were that they face in digital advertising. Which came back being:

  1. Latency (ok, we had that covered)
  2. Transparency (check, transparency as a feature is a given if you are enabling efficient markets)
  3. Ad fraud (interesting, let’s do some research)

Thus, Kubient began our long journey towards revolutionizing Ad Fraud Prevention.

Speed matters when everyone gets the same 300 milliseconds.

The Beginning

Shortly after launching our initial auction software, The Kubient [k]rew learned a painful lesson:

A major portion of a monthly payment from a buyer had been withheld because the quality of the inventory bought was misrepresented by the publisher. 

The buyer held Kubient responsible for resolving the issue or risk losing them as a client. Meetings with multiple ad fraud prevention vendors followed all with the same results—no way to truly stop it in real-time, only to mitigate it; a solution that was, in our opinion, inadequate, because advertisers had already spent their budget on fake people. Once again, our [K]rew took matters into their own hands and decided to build ad fraud prevention software that could stop ad fraud where it counts in real-time, before you bid on inventory and spend vs trying to claw it back later or have to pad budgets. Causing less money to flow downstream in the supply path to publishers.

Following months of research, we learned the only way to accomplish our goal, and accurately be able to prevent the losses, would be to utilize artificial intelligence at the time of auction.

This meant running an AI prediction service on tens of thousands of concurrent auctions in less than 10ms, something they weren’t sure was even possible, and definitely something that has previously never been done before.

Kubient immediately hired an AI expert, Vasanti Mahajan, and Christopher Francia, then VP of Product took over as project lead.

Vasanti went to task to first understand the data Kubient currently had at its disposal, while Chris worked to find a software vendor that could provide the power needed to run the AI in real-time.

“Despite being a client of Amazon Web Services, we didn’t have a dedicated account manager, so when we were looking at using their new real-time AI prediction service to power this ad fraud software we simply submitted a form requesting if they could increase our predictions per second from a limit of 100 to 10,000. Literally within a day I got a call from the lead project manager of that service wondering ‘What on earth are you guys doing that you need that many predictions per second?’, so that sorta tipped us off that our request was unusual.”

Just how unusual?

Chris and his team would soon find out. After a few meetings with Amazon, it became clear there was no out of the box software that existed that could run predictions at both the scale and speed Kubient and the industry required. Peter Bordes, our Chief Executive Officer, explains,

“You have to remember AI came from a scientific industry background, so it didn’t need to operate in the subsecond realm. Even today the most common AI products out there like Siri or Alexa operate their predictions and results around 1 to 2 seconds, so no premade solution was prioritizing speed to the levels we required.”

While Chris was hitting dead ends, Vasanti was making breakthroughs on the data and had the initial model ready to be tested, which created more pressure on the team to find a software solution and pushed Chris, as team lead, to his only option left.

“It was an extremely frustrating time because we had Vasanti creating these great models that were proving they could catch fraud but we had no way to use them in the auction. We never intended to build our own prediction software, but once you have exhausted every other avenue, what other choice was left?”

It was a long process but after twelve months of dead ends and bugs, the “Eureka” moment finally came.

From Concept to Reality

The team finally got to a spot in the software wherein the controlled environment it didn’t break anymore and the speeds were within the range they had targeted.

But, this was still in a controlled environment dedicated to just this software, so when they first decided to put it into the production auction system, they were pretty confident it would crash and all sorts of things would break on it and then would need to go back and analyze the data and test again.

But in fact, when it launched on the auction platform in October 2018, it blew away all expectations.

“When it turned on and didn’t crash, and then we saw the data on its speed we thought the logger must have been broken, but we went over everything and checked the results of all the predictions and sure enough, it was working.”

With the software tested and working, Our Founder and then Chief Executive Paul Roberts, got to work figuring out the next steps.

They’ve proven they have the ability to do it, now how can it actually be applied?

How can it be used by others for the greater benefit of the industry?

Like all things, it started with lots of testing.

Kubient reengaged with a few other ad fraud prevention companies and signed limited engagements to have those companies’ software scan our auction results. The team then took the third party’s result and compared it to their own software’s results.

“The approach of the industry and the approach we took were very different; however we needed to show our results were at the very least as good as the other guys.”

The results were clear, Kubient’s fraud prevention software caught everything the others did and much the others missed—but Kubient’s software, unlike the others, was catching it during the auction.

Chris explains why this matters so much, especially to the advertisers paying for each bid:

“Where you catch the fraud matters a lot. The systems in use today catch fraud after an ad is served, which is damaging to a lot more than advertisers wallets, because they optimize their ad spends in real time based on KPIs like impressions, clicks and conversions—the same KPIs that fraudsters are manipulating.

So advertisers are getting hit twice: they serve ads on this fraudulent inventory, then optimize their algorithms on that data, resulting in them serving ads to more fraudulent inventory. It’s a vicious cycle.”

A vicious cycle that somehow, Kubient had found a way out of.

KAI [Kubient Artificial Intelligence] ad fraud prevention is born

To understand what makes KAI truly disruptive innovation, it is important to understand how the industry generally did fraud prevention. While other vendors claim AI and Machine Learning power their technology, those terms cover a wide range of effectiveness.

“If I told you I was a basketball player, and nothing else, it could mean I played pickup at the local gym or that I was the starting forward on the Orlando Magic; the truth lies in the specifics.”

And the specifics of KAI and what the rest of the industry is doing, could not be more different. 

The industry had long taken a human driven approach to fraud prevention, while KAI on the other hand was a hybrid, utilizing both the human element and the machine element in deciding what was fraud.

“When ad impression data is generated, these companies take that data and give it to a bunch of humans who analyze it and say if it is fraud or not fraud. Those results get put into a machine and the machine feeds that data to lists that are used to spot the fraud next time it appears. It is all reactive, and isn’t a new tactic—it’s the same type of ‘AI’ that was prevalent in the 80’s.”

Breaking down even further, machine learning also operates in one of two ways:

There are two kinds of Machine Learning: Supervised and Unsupervised. Supervised relies on a human telling the machine what something is, while unsupervised is telling the machine the concept of something and then it goes and decides what fits into that concept.

This unsupervised machine learning is what really separates KAI from the rest of the pack. Enabling KAI to have a 360-degree view of the digital advertising ecosystem vs the single lenses of other solutions. The first indicator of this came in October of 2019, when KAI started flagging traffic as fraudulent that baffled the team.

“We found it really odd, because we looked at the data manually and it looked like clean traffic, so we thought it could be a false positive. But as we dug deeper, we discovered that it was actually a new kind of fraud, and the machine knew it before we did.”

This new fraud was called synthetic traffic or the SynthNet, which was a type of automated traffic that originates from a centrally controlled system instead of a network of bots. The KAI team anticipates this as just the beginning, and KAI will catch more and more fraud schemes as they happen.

“It’s only going to get smarter—the more data it sees, the more patterns it analyzes, the more powerful and accurate it becomes.”

KAI is also revealing even more secrets. As the team has fed more third party data into KAI, it has seen some troubling statistics for the industry as a whole.

In Q1 of 2020 the team ran statistical models of random samples of auction data collected across the industry. What they discovered was a fraud rate of 3.7/10, which meant 37% of the auction traffic was fraudulent. This places the fraud rate higher than previous reports but explains why fraud losses have continued to increase year over year.

“If the industry standards are catching 19% of the fraud, but the real number is 37%, plus you know that fraud losses have never slowed down despite the industry’s best efforts, then the only logical conclusion you can come away with is that the industry net has a few big holes in it so to speak.”

With the drastic impact that increased fraud will have on the industry, Kubient firmly believes that everyone should be able to have access to KAI. As a general rule, when there’s less fraud in the ecosystem, there’s more trust between publishers and partners, and everyone benefits.

Kubient currently offers two versions of KAI today:

KAI In The Audience Cloud Marketplace

Supply and demand partners who are integrated and using the Audience Cloud Marketplace automatically have access to KAI as it’s built into our transparent environments infrastructure. Partners can also access KAI directly as a standalone enterprise and self serve “lite” solution for the many companies that can’t afford the cost of the larger platforms. We believe ad fraud prevention needs to be accessible to everyone in the industry in order to stop fraud where it counts. Before it happens.

KAI Standalone

The full feature software provides the most sophisticated machine learning algorithms and is designed to satisfy the needs of any client.

Latest Insights

Kubient Acqui-Hires Team from Leading Digital Media Agency MediaCrossing Inc. to Bolster Managed Solutions Division
Kubient Doubles Down on AI and Technology Offerings, Hires Mitchell Berg as Chief Technology Officer
Kubient Partners with Verve Group to Increase Transparency and Reduce Fraud in Advertising Supply Chain

Ready to transform your advertising?