Entropy Based Invalid Traffic Detection at Scale

The second law of thermodynamics states that the tendency in any isolated system is to slide towards a state of increasing randomness i.e. entropy.

70 years ago Alan Turing combined the Entropy theory from Thermodynamics with compute power and helped win the second world war by famously cracking the Nazi enigma code, saving the lives of millions of people in the process. Proving the viability of the entropy method  in solving even seemingly impossible problems.

In 1948, Claude Shannon “The Father of Information Age” formalized Turing’s invention in to a simple mathematical formula:Since then Entropy method has been widely used in fields with vast amounts of data, many unknowns and where accurate predictions are important or critical. Application of the entropy method range from algorithmic finance, to mass-surveillance, information security and healthcare.

Nameles is the first entropy based invalid traffic and traffic anomaly detection method and is an open source initiative with years of development work already invested by a community of researchers, non-profits and academics.

– the first ever open source verification solution
– the first 100% transparent verification solution
– the first media format agnostic verification solution (banner, video, in-app)
– the first device agnostic verification solution (desktop, tablets, mobile)

In actual large scale real-time use, Nameles have been proven to have a very low total-cost-of-operation (TCO) [1][2].

Learn more about entropy here.

[1] Entropy Method for Detecting Invalid Traffic at Scale
[2] Economics of Invalid Traffic Filtering (published in feb-17)

Why Open Source

Open source solutions have been proven as a key success factor in solving internet traffic problems similar to invalid website traffic; both email spam detection and network security fields largely depend on a handful of widely adopted open source solutions. Key benefits of wide open source adoption include:

– increased transparency
– positive network effect
– improved vendor profitability
– encourages knowledge sharing
– creates a community

In the recent World Federation of Advertiser’s Compendium of Ad Fraud Knowledge “embracing open source solutions” was listed as one of the highest impact initiatives for the industry to adopt to counter ad fraud. WFA’s membership represents 90% of the $700 billion per year media investment and its guidance have been shown to be the single greatest contributor to changes in advertiser attitudes and behavior change.

 “It is also incumbent on the broader industry to accept the need for change; to put aside vested interests and embrace the potential for open solutions

– Stephan Loerke, CEO of World Federation of Advertisers

 

The paper also highlighted how success in solving similar problems, such as network security and email spam, had been largely dependent on availability of open source solutions. In the case of Network Intrusion Detection System, the leading open source solution Snort has an estimated 60% market share, and is used by many of the world’s largest companies, cybersecurity vendors, US State Department and US Military bases.

 

“Fighting video ad fraud needs to be an open source solution”
– Brett Wilson, CEO of TubeMogul

 

Benefits

Adopting Nameles provides an x-ray vision to the programmatic ad market, creating both direct and indirect cost benefits. These benefits range from significantly reduced IT overhead, to increased buyer trust, and improved optimization capability through cleaner data.

Unparalleled invalid traffic detection leads to:

– Reduced cost
– Higher profitability
– Increased valuation
– Less legal risk (from exposure to fraud)
– Higher trust from buyers
– Better energy efficiency

Savings from just reduced IT cost versus Nameles total-cost-of-operation has been proven to be at least 100:1, where for every $1 invested in to operating Nameles, $100 dollars is saved in IT cost.  A typical adtech company spends up to 35% of revenue to commissions, IT and other traffic volume related cost items. Cost-effective pre-bid filtering provides an instant remedy to profitability, and particularly to valuation (Net Present Value).

Using the the financial statements available for 7 major Demand Side Platforms, we’ve created a Net Present Value (NPV) model that proves how many unprofitable DSPs could make their business profitable simply by adopting Nameles and implementing it as intended. Below graphic shows an adoption scenario with 16% overall filtering rate for one such company.

Get the full details on the model here.

Get Started

Download the Linux Package

Nameles is currently available for Debian and Ubuntu systems.

Get the Source Code

The source code is available as a self-contained fully functional solution with SQL backend. Port to SPARK and other other common backends are in the works.

Read the Papers

Three academic papers that outline the system design, the entropy model, scalability and other key aspects of Nameles.

An entropy-based methodology for detecting Online Advertising Fraud at scale read here

Can Intermediaries in Programmatic Advertising Obtain Economical Benefit from Invalid Traffic Filtering!? Why and How? read here

Nameles: A system for Real-Time Filtering of Invalid Ad Fraud at Scale read here

Read the Manual

We’re doing our best to make sure that everything is answered in the manual, so it’s a good idea to read it before implementation.

Community

 

 

If you have a question or need help getting started, join the public Nameles Gitter channel using your Twitter or Github account with two clicks from here.

 

 

If you have an idea for improving Nameles or found a bug, report an issue or make a pull request on the Nameles Git.

 

For everything else, send email to foundation@nameles.org or call +1.585.502.8985