I wrote about the basics of how DSPs, SSPs, and Ad Exchanges work from a high level and the value they add to the marketplace in my last post – for this piece we’re going to dig into the nitty gritty of how they work from a technical perspective.
In my explanation of third-party ad serving, I outlined a 12-step process to get from publisher ad call to marketer ad creative. In an Exchange environment the process is basically the same, but with three new parties involved: the SSP, the DSP, and the Ad Exchange itself, which all act as the ad selector, instead of the publisher’s ad server.
You can view a diagram of the ad serving process at the top of this post – the numbers in the text refer to the steps labeled in the diagram.
When a browser navigates to a publisher website (1), the publisher’s web server sends back a bunch of HTML code (2) that tells the browser where to get the content (3) and how to format it. Part of the HTML code returned to the browser (4) will include a coded link known as an ad tag. That part is the same as in regular third-party ad serving. But instead of returning a DFA or Atlas tag, the Publisher Ad server will return a tag that points to a RTB-enabled SSP, typically through a dynamic Javascript tag that passes information like the publisher’s ID, the site ID, and ad slot dimensions.
From there, the user calls the SSP server (5) where the SSP reads that user’s SSP cookie ID, likely already on their machine. Assuming the user already has that SSP’s cookie on their machine (and most users will, given that the largest SSPs boast 80 – 90% reach rates to the US internet population), the SSP starts the auction (6), by requesting bids from a host of demand sources, the DSPs (7). If the user does not have an SSP cookie on their machine, their ad inventory can technically still be auctioned, but since nothing is know about that user, the price will be very low and more related to the site context than the user’s attributes. For the DSPs to truly value the impression though, they need to know something about the user that is going to see it. This is where the SSP cookie ID comes in – packaged with the bid request is the SSP’s cookie ID, along with the url the impression will deliver on, and what the current user’s frequency is on the site.
All these factors help the DSP value the impression. First, through a rather complex cookie-synching process, DSPs are able to match the SSP’s cookie ID to their own cookie on that user, which is tied to a huge cache of marketer data and 3rd party data. What kind of 3rd party data? Using the cookie ID, the DSP will be able to know if that user recently priced out a car, is flying to Paris in the next 90 days, was recently shopping for shoes, and even more general demographic information about the user such as their age, gender, income range, credit score, and much, much more. I’ll cover how that all works in a later post. But suffice to say, rich data is far and away the driver of higher bids, and the cookie ID is the mechanism through which data is associated to a user.
In addition to the cookie though, where the ad will appear, the URL, is also important. Many brands don’t want their ads to appear on just any site, even if they want that user. If the user is on a site with PG-13 content, for example, the advertiser might bid a lower amount or not at all. Similarly, the frequency of that user to the site they are on is also important to valuation. Advertisers are willing to pay a premium to reach users on their first or second pageview on a site vs. their 50th pageview for the simple fact that users are less engaged with site content and more likely to respond to an ad during their first few pageviews.
Using those pieces of data, the DSPs all value that impression and submit a bid back to the SSP (8) as well an ad redirect to send the user should their bid win the auction. The SSP picks the winning bid (9) and passes the DSP’s redirect back to the user (10). From here the process is basically the same as third party ad serving – The user calls the DSP (11), the DSP sends the user the marketer’s ad server redirect (12), and user calls the marketer’s ad server (13) and the marketer serves the user the final ad (14). Whew!
{ 32 comments… read them below or add one }
How are the workings of a mobile ad exchange (e.g. mobclix, adfonic, Opera) different than that of online described by you in your above post?
Hi Bob,
It’s a good question, but unfortunately I’m less well versed in the mobile space. My understanding is that Mobclix would work very much the same way I’ve outlined in the redirect path for RTB exchanges since it is also an exchange. Adfonic looks to be more of a traditional network however, so I would expect them to have more of blunt approach to ad pricing that is less related to impression by impression valuation due to audience characteristics, and more related to content adjacency.
Do you have any links to empirical evidence or hard data to support your claim that users are more likely to respond to an ad during their first few pageviews as opposed to 40th or 50th pageviews?
Hi Dan,
I think the best article I’ve seen on this topic is something Mike Nolet, the CTO of Appnexus, wrote about on his blog, MikeOnAds.com, way back in 2007 called Maximizing Network Revenue. While the graphs and data points he cites are technically fictitious, I can tell you from my own experience that they absolutely hold true and buyers will usually be pretty transparent with you that they will pay more for the first impression, regardless if they are buying on RTB, or directly via roadblocks or first-impression products that command a premium. You can also look at Slide 12 of this AdMeld presentation, which has a graph of fill rate on their system by frequency for further evidence that there are more buyers and thus a higher value to the first impression a user sees on a site.
Hope this helps,
Ben
Great Article Ben,
Very curious to know about how the SSP cookie gets matched to other cookies that the DSP has. You mention cookie ID/stamp?
Great if you could provide some insight into how this works, especially keeping in mind re-targeting campaigns where an existing cookie pool is leveraged to target the user on different sites.
Also it helps of you can put into perspective the Exchange / SSP / RTB platforms , are these all the same thing? Is that then saying RMX is similar to pubmatic in some way?
Thanks Sid,
Great question – the answer is a bit long for comments, so I’ve published the answer under a new post. You can read it here: http://www.adopsinsider.com/ad-exchanges/ssp-to-dsp-cookie-synching-explained/
As to your second question, the Exchange provides the liquidity to RTB auctions. Right Media Exchange (RMX) is an ad exchange. SSPs like Pubmatic optimize the demand that comes through an exchange and bids in RTB with other sources of demand, such as that from an Ad Network that may not bid in real-time, or even direct-to-publisher remnant deals. Think of an SSP as an Exchange-Plus, and as a service for Publishers only, vs. Advertisers and Publishers, which many see as a conflict of interest with both parties. Publishers can load in competing sources of demand and have an SSP run those sources against a real time auction to maximize the amount of revenue from every impression. SSPs also provide better reporting and analytics than you would find from an Exchange.
Hope that helps,
Ben
Ben. Loving these posts. Never seen anyone to go into as much detail as yourself. Nice work.
Thanks Ciaran, appreciate the positive feedback!
Great post:)
hi,
can you explain briefly about how the cookie synching happend?
or give some useful links to learn about cookie synching.
Hi Jagan,
I actually wrote a separate post about cookie syncing – you can read it here.
Ben
Excellent Ben. Thank you for the clarification above…invaluable.
You segmented above into some discussion around SSP reach rates and the lack of value in the auction in absence of SSP cookies. I would very appreciate some color on this process or directions to research around this issue. We are trying to compile real reach rates of SSP’s, facts and figures around this weaker inventory, and what the resulting process looks like when these impressions gets auctioned or routed through the DSPs.
Best,
Oren
Hi Oren,
That’s correct – without a cookie match of some sort, it is difficult for bidders to value the impression. Based on what I’ve seen, a synced cookie tends to monetize at 25 – 100% higher than a non-synced cookie, depending on the DSP. In terms of reach, I would challenge you to consider that metric in a few different ways. When most people talk about reach, they mean unique users. Every SSP will tell you they can reach 90%% of the US population, about 180MM cookies in the US, and about 400MM cookies worldwide. This is the same number the DSPs and the Ad Exchanges will give you. In essence, everybody says they can reach everybody, and actually, that’s probably true since all the major DSPs are connected to all the major SSPs in a symbiotic relationship to accomplish the same goal – make it as easy as possible to retarget cookies. The numbers can vary somewhat depending on the press release you read, but at that scale, the answer amounts to ‘everyone’.
From my perspective that means reach isn’t really a point of differentiation – inventory quality and frequency are what truly matter. Frequency is likely fairly easy to measure, but you’d have to ask the SSPs individually to get the information – you can get a bit of a sneak peak at Rubicon Project however, they actually tag their entire network via Quantcast. Some advertise the number of impressions they serve per month, but realize that they most likely don’t reach all 400MM users each month, so you can’t just divide total imps served / 400MM – that’s just how big their cookie database is. In fact, most DSPs and SSPs probably see less than 50% of their total cookies in any given month. Even if you could get the true imps served in the month and the uniques seen in a month, there’s likely to be a huge range of average frequency per user, so you really want the mean and median. And generally speaking, the higher the frequency you find, the lower quality the inventory is likely to be.
As I’ve circulated in a few posts on Twitter, I believe 90%+ of the inventory available on exchanges today is what most would consider low-quality. By that I mean it is on no-name publishers, torrent sites, below the fold, adjacent to user generated content, on social media sites, community forums, or other places most advertisers would not place a direct to publisher buy. That means that the SSPs have to have quite a bit of those sites as clients and they’re unlikely to volunteer that information. Brand publishers on the other hand you can get a sampling of on each vendor’s website. Anecdotally, I believe Rubicon is the largest from an absolute impression standpoint, Admeld after that, and Pubmatic to be the smallest. From a quality standpoint it is very difficult to say, each SSP has a handful of very high quality clients, and also likely a long-tail of lesser quality clients.
Hope that sheds some additional light on things –
Ben
Thank you so much for the quick reply Ben. For some reason I was not notified that a response came back into the forum…very nice surprise when I came back around!
It was like coming home in the 90′s to a full answering machine…
In terms of the valuation of an impression, we are certainly trying to understand that very observation of 25% – 100% delta. There is much fodder there to discover. Your comments about reach is helping us think about our problems with better depth to the term. That is always good, thank you. I will go back and dig up some of those old twitter comments as well.
My concern is how “real” that reach translates into creating targeted audiences vs what end up as low quality impressions because of the lack of data points in sync failure . It’s a salient problem, exacerbated by the 90% figure you noted, because you drop lower down the tool set, past contextualized value, and into the bottom of the barrel. I will keep you updated if I think I find something worth sharing.
Thank you again for participating in the discussion. It is a great thing your doing here.
Best,
Oren
Hi Ben,
In the Key you refer to Cookie Stamps / Requests with a Grey Line but this is not visible in the Diagram. Please can you clarify?
Thanks,
Dharmesh
Hi Dharmesh,
You’re right, I missed that – I’ll have to correct later, but that path would be parallel to Step 5, and run between the user and a longer list of DSPs, some of which might be taking place in the auction, and some that might not. If you view the source on each script I reference at the bottom of my post on DSP to SSP cookie syncing, you’ll see that some scripts call out to 15 or more 3rd parties, and I can tell you from experience that there is almost never that kind of bid density on any impression out in the marketplace.
Hope that clarifies, and thanks for alerting me to the error.
Ben
Hi Ben
Great overview of the technical and architectural aspects of SSP/DSP/AdExch/RTB. Thanks! Does your expertise extend to how DSPs assign value to arrive at an appropriate RTB for a given (single) user impression. If yes, can you please shed some light on this aspect of this process. If not, can you point me to a reference which explains the pricing? Many thanks.
Hi Stuart,
Thanks – in terms of pricing mechanisms, there are basically two methods. The first is where advertisers simply enter a flat price they are willing to pay, and the DSP submits that as the bid price for every impression. That’s pretty easy to understand. The second is an optimized method, where each impression is evaluated and valued on a host of factors. Within optimized bidding, most DSPs allow you to pick the metric for which a campaign is optimized. For example, you might what to optimize for CPM (the lowest cost per impression) for some campaigns, but CPC, or CPA (the lowest cost per click or conversion) for others. For either goal, the DSP will conduct a sampling period that varies in length to reach some statistical significance against a variety of factors so it can then predict future performance and bid more aggressively against impressions with the predicted ideal qualities for that campaign. Those factors might be inventory quality, time of day, geography, operating system, frequency, or include other pieces of available data. Each DSP will have its own special sauce to reach a decision, but MediaMath, one of the leading DSPs, actually exposes a bit of their process to clients in the UI. They have an interesting video up on their site that gives you a peek into what factors their system evaluates, and how they reach a decision. Interesting stuff: http://www.mediamath.com/products/ and click on “Brain Visualization”.
Hope that helps,
Ben
Ben,
Thanks a bunch for this, it’s definitely helpful. I have a quick question, how does it work when the publisher sets a floor price and that floor price isn’t met? Does the ssp/exchange redirect the user back to the publisher ad server and tell them that the floor price wasn’t met? Or does the ssp/exchange somehow know which reservation or house ad to show if the floor price wasn’t met?
thanks!
–Gabe
Hi Gabe,
Every SSP or exchange will require some sort of ad of last resort, often called a ‘default ad’ even if there isn’t a set price floor. That ad of last resort could be a house ad, or it could be a call back to the publisher, allowing them to re-decision the ad call with their own ad server, and host those final assets themselves. The best solution is usually for the publisher to hand over the default ad assets to the SSP / Exchange vs. getting called a second time. There are a few reasons for this – first, typically the publisher is using the SSP / Exchange as the ad tag of last resort anyway, meaning there is nothing more valuable to serve at that time from a direct sale, so unless they are hosting many house ads or changing them out on a very regular basis, having to decision the ad twice is redundant. Second, all ad serving carries a cost, even if it is a nominal cost – receiving that call again costs the publisher money. Third, decisioning an ad twice creates duplicative reporting in the publisher’s ad server, inflating inventory figures. The user might only receive one ad, but the ad server counts it as two – once as a redirect to the SSP / Exchange, and a second when the SSP / Exchange calls them back. Finally, if the publisher isn’t careful, they create the potential for an infinite loop – where the publisher ad server calls the SSP / Exchange, which calls them back, which calls out again to the SSP / Exchange, which calls them back, and so on until the ad just fails or the user abandons the page.
Hope that clarifies!
Ben
This is great stuff Ben, many of us are in sink or swim positions on a very fast moving river, thanks for your time answering our questions.
This may be a more basic question, but I am a bit of newbie working largely in the mobile ad space right now.
How do the cookie values find their way into a std OpenRTB object?
Without cookies in mobile I presume this means all requests are treated as those without a sync, this places much more dependence on the app, device and segment object, how are these populated?
Thanks Mark, appreciate your comments.
As to your question, I’m a mobile newbie myself, but I’ll try to give you as much information as I can. While many mobile devices actually do accept cookies, you’re correct in the sense that 3rd party tracking cookies like you may be familiar with in a desktop setting don’t really exist at scale in a mobile environment. Instead, device ID tends to be the user-level tracking standard for mobile environments. Typically publishers or content owners can pull or request this ID from a device with a bit of action script (see this site for more detail: http://www.pocketmagic.net/?p=1662), and then pass the information in a URL string to 3rd parties. Google actually started doing this for their AdMob network in April – http://www.clickz.com/clickz/news/2044496/google-readies-behavioral-ads-mobile-apps A best practice is typically to hash this ID to anonymize it before trying to pass it to any 3rd party, as the device ID is actually considered personally identifiable information because it can be tied back to a real person and is not delete-able by the user. On AdMob, Google anonymizes the device ID. In an app environment, I believe you would simply need to run this action script and then use a software development kit (SDK) to connect with a 3rd party system’s API. You might read over the AdMob SDK Developer’s guide, which details how to pass device ID in the spec – http://code.google.com/mobile/ads/docs/
I would also suggest you read up on the OpenRTB Mobile Specifications here – http://code.google.com/p/openrtb/downloads/detail?name=OpenRTB%20Mobile%20RTB%20API%20-%201.0.pdf You’ll see that the specs show that no device / user IDs are required by the exchange, but are optional values. At the same time, the specs allow those types of values to be passed when available, and can get pretty specific. Generally speaking though, from what I’ve heard there isn’t very much user level targeting on the mobile web right now – it tends to be focused on location, which you can get from the GPS coordinates on the phone, which again, you can call with action script from a phone, and pass to 3rd parties using a URL string on the web, or through an SDK in an app.
If you have any mobile network / technology partners, ask to speak to their developers for more information, who can help you dig in.
Hope that was somewhat helpful, if you find any other sources of information please send them over, I’d love to understand this better myself!
Ben
Thanks for the reply Ben, I am digging into it and will share whatever I find.
Hi Ben ,this post is very helpfull!
I have a technical question. In the RTB scenario where many players interact between them, how is it possible that the RTB communication work quickly?
I mean, we know that the response time to show an ad is critical to optimize inventory in the sites, because the user could spend a short time in the page and this time may not be enough to show an ad.
Could you please explain a little more about how this communication work? or give me a sort of links so I can get more information about this?
Thanks !
Hi Hernan,
Glad you found it useful – as to your question, I’m not entirely sure what you’re asking, but it seems like it could refer to a few things.
First, the bidding process may involve multiple demand side parties – lots of people are bidding on the same impression – but the bid request and bid response process happens concurrently between parties. The SSP doesn’t send a request for a bid to one DSP and then request another bid once it receives the first, it asks all the DSPs at the same time, and gives them the same short window of time (20 – 120ms) to respond. The decision process and ad serving process doesn’t take too long – perhaps 20ms per leg. All told, you want to aim for about 120ms in latency or better through the entire process, though you might expect as much as 400ms in some cases. As you might expect, 3rd party discrepancies – the amount of impressions you track on your local ad server and what your RTB partner reports as served impressions can be different by 10 – 15% as users drop out of the process before it can finish.
Secondly, each bidder already has all the information it needs in order to value the impression, even if it is using 3rd party data to inform that bid, because any 3rd party information on the cookie viewing the impression is already synched to the DSP cookie in advance, in a server to server data pass which I talked a bit about here: Syncing Your Online Data to a DMP. That’s specific to a DMP, but the process is exactly the same, but with the DSP acting as the DMP client in this case, and the 3rd parties acting as the data sources, so in place of the order management system for example.
Hope one of those answers your question – if not, feel free to write back with more specifics.
Thanks,
Ben
Hi Ben, first of all, thank you for the quick response, and for sharing all your knowledge. You were very clear in your response, but I want to go a bit more deeply than that.
When you said that the SSP sends a request to lots of DSPs, is that communication a Server-to-Server connection or simply a query through the web? Also, I assume that the SSP ensures a fast response with the timeout that you mentioned, right?
On the other hand, once we have the value of the impression, you said that the bidders have the necessary information to value the impression. I assume the information is standardized in order to get the same valuation between all DSPs. Also, I imagine that some information would be the age of user, location, etc, do you have a paper that talks about this valuation more deeply?
Thanks Ben!
Hi Hernan,
The SSP to DSP communication works through an API connection, so server to server. The reason being is that each party needs to pass certain variables to the other, so there needs to be a standard framework for how each party sends to and digests data from the other party. Each DSP and SSP operate slightly different, so there is a bit of a manual implementation process required. And yes, the SSP controls the time limit around the bid responses – if a DSP can’t respond in time, the SSP will simply conduct the auction without a bid from that DSP.
In terms of valuing the impression, not much is really standardized. In fact, given the exact same data, I would be surprised if two different DSPs returned the same bid response. On the 3rd party data side, there’s no requirements around data quality – one data provider might return different demographic information on an individual cookie as another data provider. Each data provider sources their information from different publishers, and then segments and qualifies that data into end products slightly differently.
For more information on how DSPs use data to bid at the impression level however, I’ve found this Media Math presentation helpful – http://www.mediamath.com/products/ and click on “Brain Visualization”.
Hope that helps,
Ben
Hi Ben,
This post is amazing! I’m a newbie to this industry and this post helps me a lot. Thank you very much.
I have several questions to understand ad mechanism more.
1. If no SSPs or DSPs are used and only Ad Network is used, is it correct to change the above diagram as follows?
- change “SSP / Exchange” to “Ad Network”
- delete “SSP / Exchange RTB Auction”, “DSP1-3″, “Ad Network”
2. In mobile ad space, especially in smartphone apps space, are there many cases that there are no “Publisher Ad Server” と “Marketer Ad Server”?
If that’s the case, does “Ad Network” solely handle ad information?
3. Is Akamai’s Content Distribution Network working as an image file delivery infrastructure in mobile ad space as well ?
4. I understand that Anonymized device IDs are generally used in mobile ad space. Is this same in smartphone ad space?
5. How can anonymized deveice IDs are matched between SSP and DSP? Is it same as cookie-synching you wrote in the following post?
SSP to DSP Cookie-Synching Explained
http://www.adopsinsider.com/ad-exchanges/ssp-to-dsp-cookie-synching-explained/
Great article, and good drawing. Couldn’t find step (6) in your 12-step process description?
Hi Tom,
Hm, it looks like I excluded that for some reason, thanks for letting me know. Practically speaking, step 6 and step 5 are effectively the same thing. The browser gets redirected to the SSP, which conducts the ad auction process through one of the ad exchanges, which just provides liquidity to the process. If you had a specific question around any part of the process let me know and I’ll try to clarify.
Ben
Hi Jun,
Thanks for writing in! Let me see if I can help answer some of your questions –
1. Not necessarily – you can still use an SSP to optimize revenue between ad networks only, in fact, that’s how all the SSPs originally came to be. They were all founded to optimize ad network demand, before RTB existed. The idea is that the SSP will get the publisher the most money from any combination of DSPs and Ad Networks. Many publishers will have 20 – 30 DSPs bidding on their inventory and a handful of ad networks, all optimized through an SSP platform. That said, the ad network may be optimizing various advertisers itself for its own benefit, but since the ad network doesn’t bid, the SSP usually just ends up optimizing on a blended, effective rate. The benefit to an SSP with a network however is that it can better predict the value of a given impression by ad network , and easily transition from one network to another once a network starts to pass back impressions and etc.
2. I’m not sure about this – in my experience there is always a publisher ad server of some type, even if it isn’t specifically a mobile ad server. Typically, an SDK is in place to allow a 3rd party, like a publisher ad server, or a marketer to control an iFrame in the app, but those ad serving systems are still in place to decision the ad. If the publisher is only working with one network, I suppose they could simply hardcode the call to the network, but I don’t think this is standard for most major content publishers, since most will want to work with more than just one network, and have to have a system to manage that process. Perhaps smaller app developers work this way, however.
3. Yes, Akamai was serving as a CDN for mobile as early as July 2010.
4. I don’t have a great answer for this – some parties have been using device ID (UDID) or the IMEI number as a unique ID for smartphones, but Apple is starting to mask those values in OS5, and there are clear privacy concerns about utilizing a value that the user cannot opt-out of, or control. Cookies aren’t really a suitable option on mobile phones, even if most smart phones can technically accept them, because the cookies are entirely wiped between sessions, which defeats their use altogether. So there isn’t a clear path forward here, at least on mobile web. In the app environment, publishers can force users to register, and use that registration as a way to re-identify the user, and the data they have about that user. This is useful to a degree, but still doesn’t solve the problem of allowing marketers to match the data they might have to that user. The value that cookie-syncing creates in the desktop environment really doesn’t exist, at least not yet, on mobile. Most impressions are instead valued based on CTR, and geo-location.
5. Yes, that’s it exactly – cookie syncing is how that happens.
Hope that helps!
Ben
Hi Ben,
Thank you very much for your reply and detailed/clear description.
For my understanding, I have created a diagram of non-RTB ad networks version.
https://picasaweb.google.com/lh/photo/h6r1d7pR5xAZr_VjzreePA?feat=directlink
I’m afraid this is in Japanese..but I guess you understand easily..
Thank you again!
Jun