A/B Testing using NGINX

So there are application changes which have to be tested only in production with a cohort of real users. However you may not be comfortable deploying these changes to all your users and this is where A/B testing comes in handy. With A/B testing a certain portion of your users would be seeing the new version of your application while the remaining users could continue to see the older version. As can be deduced there is no limit to the number of versions you can direct your users to. This article provides a quick overview of the mechanisms provided by NGINX to enable A/B testing.

Methods provided by NGINX
There are 2 methods :
1) split_clients
2) sticky route

Method 1 is supported by both the free and paid versions of NGINX while method 2 is supported only by the paid version – NGINX Plus.

Let’s delve into the split_clients method.

SPLIT_CLIENTS
In the NGINX configuration file you define a split_clients block which sets a variable that decides which upstream server services the client’s request. This results in a randomized assignment.
This variable is set based on a hash of some property of the client say like the IP address or one or more variables in the request URI.

Let’s look at the code block below :

In the block above the token variable in the URI is hashed and is then used to determine the value of appversion. 95% of the requests are expected to be directed to version_1a and the remainder to version_1b.
Assume that the possible range of values is 4,000,000,000 and the hash value you obtain is 1,000,000,000. This falls within the 95% range and hence the appversion variable is set to version_1a.

Let’s now see how this plays out with the upstream and server blocks.

As can be seen from the blocks above the value of the appversion variable decides which upstream servers the client requests are directed to.

Let’s now look at the sticky route method.

STICKY ROUTE
This method is used when more control over the routing is required. Here sticky persistent routing is accomplished based on a static property of the client or one or more variables in the request URI.
There are 2 ways to accomplish this one at the client’s end and the other at the server’s end.

Client side routing
If done at the client’s end routing can be based on a static value like client’s IP address or any variable found in the header of the request which is unique to the client. So all clients falling within a certain IP address range could be always routed to a specific server.

Server side routing
Here the server sets a cookie or a redirect URL and the client presents this cookie or URL on subsequent visits which enable the sticky route directive to extract the value and then consistently route the request to the same server.

The following code block demonstrates how server side routing can be achieved

http {

map $cookieroute $routefromcookie {
~.(?P<route>w+)$ $route
}

map $argroute $routefromarg {
~.(?P<route>w+)$ $route
}

upstream backend{
server 182.173.23.2 route = a;
server 182.173.23.2 route = b;

sticky route $routefromcookie $routefromarg
}

server {
…….
location / {

proxy_pass http://backend
}
}

In the upstream block the sticky route directive extracts the value of routefromcookie or if that is unavailable it extracts the value of routefromarg.
If the route is ‘a’ then requests go to that server or to the other one if the route resolves to ‘b’.

However there is some additional logic that is needed to extract the route and that is where the map block comes in.
The map blocks read the cookieroute and argroute variables and then use a regular expression to extract the route and set it in an internal variable as well as in the routefromcookie and routefromarg variables which are then read by the sticky route directive to decide the routing.

Strategies to decide the method
If randomized testing is what is desired then the split_users method offers a solution. And here using a combination of variables such as a user token and a cookie to generate a hash unique to the user as opposed to using the browser version to generate a hash since if all your users use the same browser version they could all get hashed into the same server.

If you want to control which application server a client is routed to then the sticky route method is what you need. Here a session cookie can be set for each client as soon as a session is started and that same cookie can be used to continue the session on the same server when the same client sends a subsequent request.
If this is not possible then grouping clients based on geo-location is yet another strategy. Grouping clients based on IP addresses may not be a good idea since a client may have multiple devices and could end up seeing different versions of the application on their different devices.

In conclusion NGINX enables you to easily set up and execute A/B tests and even the free version can be quite effective in achieving this objective.

Reference
https://www.nginx.com/blog/performing-a-b-testing-nginx-plus/