I’ve already written a number of posts on how to use Charles Proxy for ad operations work, including how to setup Charles for mobile debugging, both over Wi-Fi and cellular networks. This post is an advanced Charles Proxy tutorial, meant to highlight best practices and advanced tips once you know the basics. I’ve suggested use cases for each that apply in the ad ops world, or that I use the tool for specifically.
Save / Open Charles Proxy Sessions
If you’ve ever been frustrated when trying to reproduce a technical error, being able to save your Charles Proxy session, send it to someone else, and have them open it on their machine to review the same issue is a godsend. I find this is especially helpful if you need to loop in the engineering team’s help, or send evidence of a particular issue to a vendor’s team. Charles can even open sessions saved in browser based developer tools as well, so you don’t necessarily need to pay for a Charles license for everyone on your team to benefit from this feature.
As a best practice, I also try to record each page load I render as a separate session, so I don’t get confused as to what happened on what refresh cycle with my browser. In this instance Charles Proxy easily outpaces a browser developer tool, which typically requires you to open a new tab if you want to load a new session and seems to consume a huge amount of system resources per session. For whatever reason Charles is much more efficient in the way it leverages RAM.
Another option on this front is to leverage the Auto-Save feature in Charles, which can automatically save your session every X minutes or so. Once Charles saves your history to a file, it clears that data from the active session, freeing up system resources. The multi-session approach makes sense if only need to test for a short period of time, while the Auto-Save feature is a better option if you’ll be testing lots of things over an hour or longer. For example, if you were validating an ad server migration across a number of websites. (more…)
Charles Proxy is a popular application for web developers generally, also used in the Ad Operations field. Ad Ops staff use Charles to debug digital ads and ad technology like header bidding setups, ad server configurations, and so forth. What makes Charles Proxy useful is that it records the HTTP requests between a computer’s browser and all the different servers it actually interacts with to render a webpage. That includes communication which isn’t clearly visible in the source code of a webpage, but the browser encounters through redirects or items referenced or embedded in other scripts.
In my own view, Charles Proxy is more difficult to learn than developer tools, and can be a little frustrating in the setup work that’s often required. New users should expect some bumps on the way – be patient. So I don’t look to it as my first option, but when I’ve really got to dig in, there’s simply nothing better.
Setting Up Charles Proxy
Depending on what you need to test, you’ll have to do a bit of initial configuration to get Charles working. Typically, you’ll need to at least install Charles Proxy’s root certificate on your laptop, and if you plan to test mobile traffic, on those devices are as well. I’ve written a thorough walkthrough of how to install Charles’s SSL certificates on your laptop and phone here (for Mac), as part of another article on how to use Charles over a cellular network. (more…)
I’ve covered header bidding from a few different angles on this site in the past few months, from how it works and helps publishers earn more revenue, to documenting how specific companies integrate with real publishers. One key element that I’ve been missing though has been the topic of header bidding wrappers, which are increasingly important for publishers, especially those working with multiple partners at once.
What are Header Bidding Wrappers?
Header bidding wrappers provide a number of key technical and operational benefits that help publishers work with multiple header bidding solutions at once. Typical and expected features of a wrapper include an asynchronous container to ensure all partners have their bid requests triggered at the same time, a universal timeout setting to manage how long the browser waits for bidders to respond, and partner specific “adaptors” that allow the wrapper to translate all bids into a common key value for the ad server.
Step-by-Step, the user calls the publisher web page (1), which responds with the site content (2), part of which contains the header bidding wrapper’s code. When the user calls that wrapper code (3), it instructs the user to make an asynchronous call (4) to one or many SSPs or other header bidding integrated partners (5). Those bidders each hold their auction as they normally would and respond to the user (6). The wrapper (which the user has already rendered) then takes those responses and formats them in the request to the ad server, which the user now makes (7). Finally, the ad server makes it’s decision and responds to the user with the final ad (or redirect). If the ad server decides to fulfill the ad with a header bidder’s line item, an additional macro in the creative tag tells the user to render the specific tag from the right bidder.
The image below is the exact creative tag you’d implement with a wrapper instead of a vendor’s. For non-technical folks, it essentially says, “if you want to render an ad, tell the user to lookup whatever ad tag was associated with the winner’s response”. It uses the pattern macro in DFP (%%PATTERN%%) which is handy to reference dynamic variables you want to pass into your ad request.
I think of header bidding wrappers as a stripped down version of a tag management solution designed for header bidding use cases. They typically don’t provide a user interface like a tag management system would, but they have a number of key technical and operational benefits that help publishers work with many header bidder partners at once, just like tag management solutions help publishers work with many different tracking pixel companies at once.
Publishers who work with more than one header bidder (andwhy wouldn’t you?) can benefit from using a wrapper solution, and there are many in market to choose from, as detailed lower down in this article.
Benefits of a Header Bidding Wrapper
Critics of header bidding tend to focus their ire on two major concerns; operational setup and latency, both of which are legitimate points. Header bidding is without a doubt a more technically complex integration to manage than a waterfall setup, and can really slow down your site if you don’t manage the possible impact on latency through a timeout setting. Not only that, but one poorly implemented partner can ruin the setup for other partners and sabotage your entire operation. Believe it or not, this is one of the reasons the wrapper was developed in the first place, for high performance vendors to protect themselves from lower quality integrations at a publisher.
Similarly, header bidding can only provide the revenue benefits of holistic yield management with multiple, and ideally hundreds to thousands of line items that can be painful to setup. Header bidding wrappers were created to help publishers manage these primary pain points.
Central Timeout – Some header bidding partners let you set a timeout, and some don’t. But, if you place every partner’s code in a central container that you can set a timeout for, then you can protect your site from hanging while waiting for a response from a problematic bidder. Even for the partners that will let you set a timeout, it’s an annoying prospect to have to manage that configuration in multiple places and makes it easier to make a mistake if you plan to test your timeout setting (which you should).
Asynchronous Container – For me, this is the primary benefit of a wrapper. Most publishers implement their ad server with asynchronous tags, meaning the user doesn’t have to wait for the ads to load in order for the site content to load. And the idea here is the same, but with an added rub. Keeping your header bidding code asynchronous to the page keeps your ad experience asynchronous, but it also ensures one header bidder can’t block another bidder’s code from loading. Think of it this way; if you have five header bidders, and four of them come with an asynchronous tag by default, but you put them below a synchronous tag, then none of those other four partners can even start their process before the first partner’s code is complete and their response has come back. This might take 1000ms or more, which puts your partners on uneven footing. Being able to put all header bidding tags in an asynchronous container solves this problem. You can use tools like the Headerbid Expert Chrome plugin (image below purely illustrative) to see how tags take different amounts of time to load, and start loading at different times, giving them a disadvantage.
Consolidate Line Items – When I looked at header bidding for the first time, my initial reaction was sympathy for the publisher side trafficker that had the lousy task of manually creating the 10,000 line items that were put in place. Now, they didn’t have to create quite so many, and these days most header bidders will help create these line items automatically through ad server APIs, but line item creation and management is still a major concern. For one, since each header bidder is using a different key-value parameter in their setup, they each need their own set of line items. That means every time you add a partner, you have a huge amount of trafficking work to do, or you have to provide admin access to your ad server to facilitate creation through the API. And at least when it comes to DFP, you can’t setup an infinite number of line items – even premium accounts are usually capped at 65,000 or so. Wouldn’t it be easier then if the partner specific key-values could somehow be abstracted into a common parameter? This way, publishers only have to setup line items once for all partners, even if they add or remove specific ones down the road. This is another key feature of a wrapper, in that it has logic that finds the highest response from any partner and injects that response into the ad request through a common key. Then, this common key is what the line items use for targeting. Add or remove whatever bidders you want in the wrapper, the key stays consistent and the line item targeting never needs to change because the wrapper provides the translation layer.
Easier to Add / Remove – For all the reasons above, a header bidding wrapper makes it far easier to add or remove a header bidding partner for ad operations. The wrapper protects you and your users from crappy technology, and your ops team from painful setup work. In other words, wrappers make header bidding scalable across many partners.
Cons of a Header Bidding Wrapper
Unfortunately, it’s not all unicorns and rainbows when it comes to header bidding wrappers and there are some downsides to using a wrapper. I personally think they are far outweighed by the benefits, but just so you know what you’re getting into:
Initial setup – there’s more setup work to do, at least initially to use a wrapper because you have to understand how to configure the wrapper and what it supports and doesn’t support. This is a minor investment in my experience, and one that pays huge dividends in the form or turn-key operational work thanks to the common key-value a wrapper can provide to the ad server.
Partner stickiness – One of the main benefits of header bidding to the publisher is that it commoditizes the SSP service and makes it easy to add or remove partners at any time. The same can’t be said, at least to the same degree with your wrapper. Strip out your wrapper and you will have to change line item targeting, change configurations, and generally have migration work to do. That makes the wrapper one of the stickiest parts of the header bidding business, and no doubt while every partner is rushing to establish their own technology as the standard.
Partner Resistance – This is the big one. As you’ll find if you start to work with wrapper solutions, many bidders don’t want you to put their tag in a wrapper. In my view, there are a few reasons for the resistance; one one hand, many companies feel like they have to effectively support someone else’s technology. If you put company A’s tag in company B’s wrapper and it doesn’t work, your first stop is likely company A, and they’re the only ones with the real incentive to get it to work. That’s pretty unpopular with the implementation teams at these various companies and can soak up a lot of time. Not only that, but should the wrapper upgrade its code or make a change that breaks company A’s integration, the whole process has to start over. Yuck.
The other main reason I see out there for partner resistance is that many bidders are suspicious wrappers facilitate data collection by the company that owns the wrapper technology. One of the big shifts in header bidding for SSPs is they no longer have perfect knowledge of why they didn’t win an impression. Were they outbid? If so, by how much? What’s the going rate for the publisher’s inventory? All that data was available in-house before to a waterfall partner, and now it’s fragmented across many companies. The wrapper presents a piece of technology that’s in a great position to collect that data and push it back to the wrapper owners, giving them a data advantage over other companies. Amazon and Criteo in particular have been mentioned in the press as requiring their solution to sit outside of a wrapper solution.
Because of this, my belief is the endgame in wrappers will go the way of OpenRTB, where the IAB defines standards of how the wrapper should work, what it should and shouldn’t be allowed to do, and makes it far easier for everyone to understand and feel like they have collective ownership over.
Who Provides Header Bidding Wrappers
There are many companies that provide wrapper solutions for header bidding, and within the year I’d expect that virtually every company that can work as a header bidding partner will have their own wrapper. Below is a list of options to consider.
Prebid.js (incubated at AppNexus): I’ve contributed to this project! Prebid.js was initially developed at AppNexus but is now one of the only open source solutions, which means you don’t have to be an AppNexus customer to use it, and that there are many publishers and independent developers contributing to the source code. As of time of writing, there have been 8 releases and 23 contributors.
Pubfood.js (incubated at Yieldbot): this is the only other open source solution in market today to my knowledge, and perhaps the most widely advertised one in market as well. As of time of writing, there have been 14 releases and 3 contributors.
OpenX’s Meta: Just announced last week, OpenX’s Meta solution is unique in that it is a server-side wrapper, which means the browser doesn’t handle the communication between each header bidder, but rather a server does. This would clearly have benefits on latency, though it would also require explicit support from partners. It will be interesting to follow this one as it has more time in market.
Contango’s (aka Technorati) SmartWraper: to my knowledge this was the first wrapper solution in the market, and is said to offer some analytics services as well. I admit I don’t know much about it.
Sovrn’s HeaderSuite: My knowledge is not strong on this solution either, but from the look of the site, Sorvn appears ambivalent on how publishers integrate with their solution and offers multiple approaches.
bRealTime’s Biddr+: bRealTime (part of CPXi) announced their wrapper solution in January of this year and has a handy matrix of features documented on their site.
Others: Based on this AdExchanger article there are supposedly other wrappers out there or in development from companies like Rubicon, and Pubmatic, but without public facing documentation it’s hard to say what they do or don’t offer at this time.
Features to Look For in a Header Bidding Wrapper
If you’re considering working with a wrapper solution, there are a few key features I’d recommend paying attention to.
Fee Structure – The open source header bidding wrappers are completely free, though they are the most stripped down in terms of support and service as well. These are well suited to more technically adept publishers, and publishers that might want to customize a solution without having to build it from the ground up. Other vendor-specific solutions typically come with a fee, which could be worth it for the additional support, analytics, and other value add services. That said, in my view it doesn’t make sense to pay a volume-based fee for a wrapper solution because there’s no decision the wrapper needs to make. The code for the wrapper should sit on a very cheap CDN service, or should be possible for the publisher to host themselves.
Transparent – If you use a wrapper, you want to make sure you have some audit rights over how it works and can validate that it does in fact provide an equal playing field. To that end, if you are using a consolidated key-value for operational purposes, you still want to be able to log each bidder’s response to your ad server as well for analytical purposes.
Are you working with a header bidding wrapper? Which one and how has the experience been? Please leave your feedback in the comments for everyone’s benefit!
Recently I was speaking with a friend who’s heading up a new digital publishing organization that’s taking their sales in-house, and they’re shopping for all the usual trappings of ad technology, as well as standing up an Ad Ops team from scratch.
At first I thought, “good luck with that!”, but then after some more serious thought, it occurred to me what a unique opportunity he had to build a world class organization. After all, so many organizations started their Ops teams so long ago, and have entrenched platforms, and business lines to support that they probably wouldn’t work with today if they didn’t have to.
Starting a new team in this day in age still has all the downsides of inexperience, but all the benefits of learning from everyone else’s mistakes. After all, how many of us in the Ops community haven’t thought at one time or another, “if I could just blow it all away and start from scratch…”, oh how we’d do things differently.
It got me thinking – what would the best Ad Ops team in the world look like? (more…)
My original header biddingarticles have been read by over 10,000 people since I posted them in late June of this year, and one of the most common questions I got in the comments and on other channels was where to see a header bidding integration live on a real publisher. At the time, there wasn’t much press on the topic, but in the past months AdExchanger, Digiday, BusinessInsider, and many other sites have posted articles, and publishers have come out to talk about their experience with the technology. So now, there are lots of examples to look at to get a better understanding of what a header bidding call actually looks like, what information it returns, and how the header bid gets into the publisher ad server.
Fortunately, using the built in developer tools in either Chrome or Firefox, we can easily see if a given publisher is using header bidding technology by searching for ad tech domains and explore the details from each call. All we need to do is check the header bidding response and the ad server request code to see what variables are passed as custom key-values (the custom_params variable specifically in DFP), which will tell us what partners are in place. You can even see how much marketers are willing to pay for your ad space. I was able to find an example integration for every header bidding integration available on the market today by looking at the sites who were quoted in press articles (specific sources are listed at the end of this post), and screenshot both the response code from the header bidder as well as the matching parameter in the call to the ad server. Hopefully this helps everyone understand more tactically how header bidding actually works, and what to expect if you implement this technology for your own site.
Example Publisher: HotAir.com
From what I can tell, Amazon is by far the most common header bidding implementation. It’s often the first one publishers seem to do, and it makes sense given Amazon offers a large amount of high value retargeting demand and is very picky where they buy, preferring private exchanges.
Amazon also has some interesting nuances to their implementation – for one, they pass a variable that combines a number of pieces of information in a single value. The structure of the variable essentially describes either a display or mobile bid, a shorthand for the size being bid on, and an encoded price value. So, ‘a3x2p9’ can be broken down into three sections: ‘a’ is the code for an display bid; in some cases you’ll see an ‘m’ instead. ‘3×2’ is the shorthand for the 300×250 ad size, which in some cases is not abbreviated but written out long form. Naturally you can expect to see ‘7×9’, ‘3×6’, ‘1×6’, and other common IAB standard values here.
Additionally, ‘p9’ means ‘price tier 9’, which refers to a rate that Amazon and the publisher have agreed upon in advance and isn’t necessarily the same value from publisher to publisher. In other words, you can’t say that a p9 value on publisher A is worth the same as a p9 value on publisher B. In this way Amazon is the one header bidder that entirely obfuscates how they value a given impression. Other header bidders integrate to the ad server using an arbitrary value like Amazon, but the exact price they are willing to pay is typically accessible in the raw JSON response. Finally, Amazon tries to be very standardized in their implementation in that they always return the same number of bids on every page, even if the specific template doesn’t have an ad for every bid.