- RTBkit will become the dominant open-source software framework designed to make it easier for technical users to build their own real-time ad bidding systems.
- RTBkit shall promote standardization, a flexible software architecture, actionable reporting metrics, and simplicity, in order to lower the barrier to entry and encourage more contributors to help develop the platform faster.
- RTBkit promotes collective ownership of the project, for the economic benefit of all the participants. Private forks diverging from RTBkit are discouraged.
- RTBkit ranks a set of values on three tiers, and encourages community participation based on the criteria outlined below. If we have to make a choice between including contributions or not, we will use the tiered system in the listed order:
- 1st priority: speed of execution; maintaining the highest QPS/operational cost ratio; operability
- 2nd priority: backward compatibility with the core stack, as reflected in the installed client base
- 3rd priority: system flexibility; experimentation with the current stack, with contributions incorporated in the main code base in order to benefit the community; high availability.
What is RTBkit?
RTBkit is an open-source software package that takes much of the hard engineering work out of creating a Real-Time Bidder by breaking the problem down into well-defined components. Its open, service-oriented architecture can be used to assemble a bidder as simple or complex as desired. The RTBkit core connects to exchanges and routes bid requests and data through a configurable set of components which can be extended to implement a customized bidder.
- Exchange Connectors allow RTBkit-based bidders to plug into any source of bid-request (exchanges, SSP’s or pre-existing bidders)
- Bidding Agents are where bidding logic is implemented. Agents tell the RTBkit core which bid requests they want to receive (i.e. which targeting filters to apply) and they decide whether/how much to bid and with which ad tag.
- Augmenters are where data can be added to bid requests before the requests are filtered and dispatched to Agents. An Augmenter can add first or third party data to a bid request, or implement part of a bidding strategy such as clearing-price estimation, frequency capping, blacklisting/whitelist management etc.
- Data Loggers can subscribe to a stream of event notifications that they can use to create log files, feed reporting systems, feed data into optimization systems etc.
The RTBkit core is implemented as a set of C++ services which run on Linux and communicate with each other using ZeroMQ, and which expose a set of REST APIs for monitoring and configuration. It is highly optimized and can deal with 20,000 bid-requests per second on a single server. Service discovery is provided by Apache Zookeeper and basic reporting is via Graphite.
RTBkit Interfaces, or “The Story of a Bid Request”
- Exchange sends bid request (“BR”) to RTBkit core
- RTBkit core relays bid request to Augmenter
- Augmenter returns augmentations (“+”) to RTBkit core
- RTBkit core sends augmented bid request (“BR+”) to Bidding Agent
- Bidding Agent computes bid and returns it to RTBkit core
- RTBkit core relays bid back to Exchange
- Exchange (or Ad Server) sends win notification (“Win”) to RTBkit core for a given bid request
- RTBkit core relays win notification to the Bidding Agent that bid on that bid request
- Click Tracker/Pixel Server sends click or pixel hit event notification (“Event”) to RTBkit core
- Clicks are for a given bid request
- Pixel hits are more generally for campaigns
- RTBkit core relays event notification to the Bidding Agent
- The one that won that bid request in case of clicks
- Any that subscribe in case of pixel hits
Data loggers can subscribe to notifications for every useful step in this pipeline: bid requests, augmentations, bids, wins (or losses), clicks and pixel hits.
Exchange Connectors translate bid requests and responses from whatever format the exchange/SSP/upstream bidder sends to RTBkit’s internal format. RTBkit’s internal format is similar to the OpenRTB bid request representation.
Exchange Connectors sit in-process with RTBkit core components and are thus implemented by extending high-performance C++ classes.
Bidding Agents’ sole responsibility is to compute how much to bid on bid requests and with which tag. They receive only bid requests which match their targeting criteria, and RTBkit will always honour exchanges’ hard-real-time constraints, so Bidding Agents are easier to write than bidders which communicate directly with exchanges. They don’t have to cope with multiple thousands of bid requests per second, and they don’t have to meet strict uptime guarantees. The driving principle behind RTBkit’s architecture is that all of the repetitive tasks that must be performed for all bid requests (filtering, computing which tags are compatible with a bid request, frequency capping etc) should be handled before the request arrives at the Agent level, to make implementing Bidding Agents as straightforward as possible, to allow developers to focus on bidding strategies and optimization.
Bidding Agents are standalone processes which are built on top of a C++ base class, and essentially they implement a single ‘bid’ method.
The promise of Real-Time Bidding is that advertisers can bring to bear all available data to make ad-purchasing decisions. Augmenters can be used to add data to bid requests, on a per-Agent level if desired, and Agents can specify filtering based on Augmenter-provided data. The canonical example of an Augmenter is a frequency-capping system: the Augmenter provides, for each Agent, information on the number of impressions that have been shown to the user whose ID appears in a bid request, via a lookup in an external data source. Agents can specify that they only want to receive bid requests from users who have seen fewer than a certain number of impressions, and they could also use the number of impressions shown to date to adjust their bid.
Augmenters are standalone processes which are built on top of a C++ base class, and essentially they implement a single ‘augment’ method. Augmenters can be written to access any data-store able to respond within the latency requirements required by the exchange.
Real-Time Bidding not only relies on data to buy advertising but it also generates data itself: bids received, bid upon, won, lost, clicks etc. Data Loggers can subscribe to notifications of all of these events from RTBkit and can they log these to disk or relay them into monitoring, reporting or logging frameworks (i.e. databases, MetaMarkets Druid, Kafka etc) as desired.
Data Loggers are standalone processes which receive event notifications via ZeroMQ.
The RTBkit core includes a sophisticated real-time budget control system based on double-entry bookkeeping which is designed to prevent overspending and to make it safe to build custom pacing logic. The RTBkit core tracks budgets in a hierarchy of accounts, and a Bidding Agent can only bid if the account it is linked to has enough budget remaining to cover winning the bid. Budget can be transferred between agent accounts and their parent accounts via a simple REST interface. A very simple pacing system could therefore be implemented by transferring small amounts of budget into an Agent’s account every few seconds, although obviously more complex schemes are possible.