Microsoft Ads is adding an additional feature to the Experiment functionality launched last July.: cookie-based audience specifications.
What is the Cookie-Based Option?
Unlike experiments that are search-based only, cookie-based audience splits means that the system will “remember” which version they were shown.
Moving forward, they will only see ads from that campaign, vs. being treated as a new user every time they search. The audience “buckets” will occur at the cookie level vs. at the session level.
Microsoft notes this is preferable over Search-based Experiments for testing items such as creative testing, since the user is only responding to one source or another and it will remain that way for their cookie ID.
How Did It Work Before?
The first iteration of Experiments was search-based.
It relies on randomization for showing the ads between the original campaign and the Experiment, and it happens every time a user searches.
There was no “memory” of the user having performed the action previously, unlike cookie-based audience splitting, which buckets audiences by cookie identity vs. what they searched in that single session.
This method means users could search multiple times and be shown ads from both the control and experiment versions.
Microsoft recommends this for testing things such as bidding methods, since the comparison would be at the bid level and is ultimately a study in cost.
How to Enable Cookie-Based Splitting
Users can access Experiments via the Experiments tab.
The “Create an experiment” module will appear. Here, users select the Campaign they want to run the test on, name it, specify the dates, and what percentage of the audience should see the Experiment.
The new feature for cookie-based audience splits is now under the “Advanced Options” section:
The Difference in Rate of Experimentation
There is a trade-off for having the potentially higher accuracy of cookie-level data:
The amount of time it will take for statistically significant results.
Here is a simplistic example:
If you have 10,000 users eligible for your Experiment:
- They search an average of 3 times
- That’s 30,000 times your Experiment would run against your control.
- It would give each instance roughly 15,000 searches of data if you run your Experiment split at 50%.
However, if you have 10,000 users using cookie-based:
- Their multiple searches don’t count.
- You would have a 50% split to each instance of only 5,000 users per each group.
- The smaller pools mean it will probably take longer for the results to be statistically significant.
The cookie-based feature is available in all accounts now. You can find Microsoft’s full announcement here, along with an overview of using Experiments here.
More Resources
Image credits: Susan Wenograd
