Targeting Random Users

Martin Strobel

Computer systems are notorious for being able to do many precise operations quickly, but struggling to do things at random. Imagine you want to A/B test two different notifications on a small set of your user base before broadcasting a promotional offer to everyone. How would you choose two random subsets of your users? How would you make sure that there was no intersection between those subsets? Finally, how would you exclude the users whom were tested on from the later notification so that they didn’t see the same promotional offer twice? Using Azure Notification Hubs tag expressions, and one simple technique, this scenario is easily and quickly accomplished.

Consider a group of 100 users, each with a unique ID from 0-99. The following C# method would choose 10 distinct users at random:

public static ISet<int> ChooseUserIds()
    var rand = new Random();
    var chosenUsers = new HashSet<int>();

    while (chosenUsers.Count < 10)

    return chosenUsers;

Assuming you tagged your installations or registrations with the appropriate user ID, it is trivial to target the 10 randomly chosen users by building a tag expression as follows:

<userIdTag1> || <userIdTag2> || <userIdTag3> || ...

Sending with this tag expression shows how easy it is to target 10% of your user base if you have only 100 users and a 1:1 relationship between tags and users. But there are very normal and boring performance considerations which limit the number of tags that can be referenced in a single expression. For that reason, this approach doesn’t scale.

But, realistically if you’re just trying to target 10 percent of your user base, there’s no need to specify each and every user that should be targeted. Imagine that as each new device is registered you assign it a tag with random number in the range 0-9. No matter how many devices were registered, you could easily target 10% of them by targeting just one of those tags. If you need more granularity so that you can target just 1% or 0.1% of your user base, that’s easily handled: just add more digits to the randomly assigned number. With just three digits in your tags (“rand904”) there are 1000 options you could choose from, allowing you to target an incredibly small audience with a single tag. Need to target 2%? Easy, create tags with 2 digits (“rand46”) and target two of them with an expression like rand46 || rand77. And there’s no need to choose one number of digits; you could apply multiple tags with differing numbers of digits to allow different levels of granularity as the situation calls for it.

This approach produces much more concise tag expressions than enumerating each individual target, and will scale with you as your user base grows.

But what about the scenario that this post began with? Where you’d like to send to two distinct small groups initially, choose the more effective message, then send to the remaining untargeted users? Tag expressions come to the rescue again.

To keep numbers simple, let’s have each test group be 10% of your users. Set A will be users who had the tag rand3 and set B maps to rand9. You can send the first two notifications by just targeting each of those tags individually. And finally, to target the remaining users you can target the tag expression !(rand3 || rand9). Simple as that, you have targeted three random, easy to reason about, sets of users.


Discussion is closed.

Feedback usabilla icon