class ABMiddleware:
...
def process_view(self, request, view_func, view_args, view_kwargs):
if should_do_a_b_test(...) and "template_name" in view_kwargs:
# Modify the template name to one of our Path templates
view_kwargs["template_name"] = get_a_b_path_for_view(view_func)
response = view_func(view_args, view_kwargs)
return response
It's useful to take a step back and abstract what A/B testing is trying to do before diving into the code. What exactly will we need to conduct a test?
A Goal that has a Condition
At least two distinct Paths to meet the Goal's Condition
A system for sending viewers down one of the Paths
A system for recording the Results of the test
With this in mind let's think about implementation.
The Goal
When we think about a Goal on the web usually we mean that a user reaches a certain page or that they complete a specific action, for example successfully registering as a user or getting to the checkout page.
In Django we could model that in a couple of ways - perhaps naively inside a view, calling a function whenever a Goal has been reached:
But that doesn't help because we'll have to add that code everywhere we need it - plus if we're using any pluggable apps we'd prefer not to edit their code to add our A/B test.
How can we introduce A/B Goals without directly editing view code? What about a Middleware?
class ABMiddleware:
def process_request(self, request):
if a_b_goal_conditions_met(request):
a_b_goal_complete(request)
That would allow us to track A/B Goals anywhere on the site.
How do we know that a Goal's conditions has been met? For ease of implementation I'll suggest that we know a Goal has had it's conditions met when a user reaches a specific URL path. As a bonus we can measure this without getting our hands dirty inside a view. To go back to our example of registering a user we could say that this goal has been met when the user reaches the URL path:
When thinking about Paths in Django it's natural to jump to the idea of using different templates. Whether there is another way remains to be explored. In A/B testing you make small differences between two pages and measure the results. Therefore it should be a best practice to define a single base Path template from which all Paths to the Goal should extend.
How should render these templates? A decorator is probably a good start - it's a best practice in Django to include a parameter template_name to your views a decorator could alter this parameter at runtime.
You could see this decorator either introspecting the wrapped function and modifying the template_name argument or looking up the correct templates from somewhere (like a Model). If we didn't want to add the decorator to every function we could implement this as part of our ABMiddleware:
class ABMiddleware:
...
def process_view(self, request, view_func, view_args, view_kwargs):
if should_do_a_b_test(...) and "template_name" in view_kwargs:
# Modify the template name to one of our Path templates
view_kwargs["template_name"] = get_a_b_path_for_view(view_func)
response = view_func(view_args, view_kwargs)
return response
We'd need also need to add some way to keep track of which views have A/B tests running etc.
A system for sending viewers down a Path
In theory this is easy but there are lot of different implementations so it's not clear which one is best. We know a good system should divide users evenly down the path - Some hash method must be used - Maybe you could use the modulus of memcache counter divided by the number of Paths - maybe there is a better way.
A system for recording the Results of the Test
We need to record how many users went down what Path - we'll also need access to this information when the user reaches the goal (we need to be able to say what Path they came down to met the Condition of the Goal) - we'll use some kind of Model(s) to record the data and either Django Sessions or Cookies to persist the Path information until the user meets the Goal condition.
Closing Thoughts
I've given a lot of pseudo code for implementing A/B testing in Django - the above is by no means a complete solution but a good start towards creating a reusable framework for A/B testing in Django.
On further reflection and investigation of Google Website Optimizer it's apparent that there are gaping holes in the above logic. By using different templates to represent Paths you break all caching on the view (or if the view is cached it will always serve the same path!). Instead, of using Paths, I would instead steal GWO terminology and use the idea of Combinations - that is one specific part of a template changing - for instance, changing the <h1> tag of a site.
The solution would involve template tags which would render down to JavaScript. When the page is loaded in the browser the JavaScript makes a request to your server which fetches one of the possible Combinations.
This way you can test multiple combinations per page while preserving caching!
Update
There still is room for template switching - say for example you introduce an entirely new homepage and want to test it's performance against the old homepage - you'd still want to use the template switching technique. The thing to keep in mind is your going to have to figure out some way to switch between X number of cached versions of the page. To do this you'd need to override the standard cached middleware to see if their is a A/B test running on the requested URL. Then it could choose the correct cached version to show!!!
Update
Using the ideas described above I've implemented a pluggable app for basic A/B testing Django. You can get it off Github:
If you use the GET parameters like you suggsted (?ui=2), then you shouldn't have to touch urls.py at all. Your decorator can inspect request.GET['ui'] and find what it needs.
To avoid hardcoding template names, maybe you could wrap the return value from the view function? Instead of returning the output of render_to_response, you could return a tuple of (template_name, context) and let the decorator mangle the template name. How about something like this? WARNING: I haven't tested this code
This is a really basic example, but I hope it gets the idea across. You could modify several other things about the response, such as adding information to the template context. You could use those context variables to integrate with your site analytics, like Google Analytics, for example.
As a bonus, you could refactor this decorator in the future if you decide to stop using GET parameters and move to something based on cookies, etc.
Update If you already have a lot of views written, and you don't want to modify them all, you could write your own version of render_to_response.
Justin's response is right... I recommend you vote for that one, as he was first. His approach is particularly useful if you have multiple views that need this A/B adjustment.
Note, however, that you don't need a decorator, or alterations to urls.py, if you have just a handful of views. If you left your urls.py file as is...
(r'^foo/', my.view.here),
... you can use request.GET to determine the view variant requested:
If you want to avoid hardcoding template names for the individual A/B/C/etc views, just make them a convention in your template naming scheme (as Justin's approach also recommends):
What happens here: 1. The passed UI parameter is stored in the session variable 2. The same template loads every time, but a context variable {{ui}} stores the UI id (you can use it to modify the template) 3. If user enters the page without ?ui=xx then in case of index2 he's redirected to '?ui=33', in case of index1 the UI variable is set to 0.
I use 3 to redirect from the main page to Google Website Optimizer which in turn redirects back to the main page with a proper ?ui parameter.
您还可以使用 Google Optimize 进行 A/B 测试。 为此，您必须将 Google Analytics 添加到您的网站，然后当您创建 Google Optimize 实验时，每个用户都会获得一个包含不同实验变体的 cookie（根据每个变体的权重）。 然后，您可以从 cookie 中提取变体并显示应用程序的各个版本。 您可以使用以下代码片段来提取变体：
ga_exp = self.request.COOKIES.get("_gaexp")
parts = ga_exp.split(".")
experiments_part = ".".join(parts[2:])
experiments = experiments_part.split("!")
for experiment_str in experiments:
experiment_parts = experiment_str.split(".")
experiment_id = experiment_parts[0]
variation_id = int(experiment_parts[2])
experiment_variations[experiment_id] = variation_id
You can also A/B test using Google Optimize. To do so you'll have to add Google Analytics to your site and then when you create a Google Optimize experiment each user will get a cookie with a different experiment variant (according to the weight for each variant). You can then extract the variant from the cookie and display various versions of your application. You can use the following snippet to extract the variant:
ga_exp = self.request.COOKIES.get("_gaexp")
parts = ga_exp.split(".")
experiments_part = ".".join(parts[2:])
experiments = experiments_part.split("!")
for experiment_str in experiments:
experiment_parts = experiment_str.split(".")
experiment_id = experiment_parts[0]
variation_id = int(experiment_parts[2])
experiment_variations[experiment_id] = variation_id
发布评论
评论（5）
在深入研究代码之前，退后一步并抽象出 A/B 测试想要做什么是很有用的。 我们到底需要什么来进行测试？
考虑到这一点，让我们考虑实施。
目标
当我们在网络上考虑目标时，通常指的是用户到达某个页面或完成特定操作，例如成功注册为用户或进入结帐页面。
在 Django 中，我们可以通过几种方式对其进行建模 - 也许天真地在视图中，每当达到目标时调用一个函数：
但这并没有帮助，因为我们必须在需要的地方添加该代码 - 再加上 if我们正在使用任何可插入应用程序，但我们不希望编辑其代码来添加 A/B 测试。
如何在不直接编辑视图代码的情况下引入 A/B 目标？ 中间件怎么样？
这将使我们能够跟踪网站上任何位置的 A/B 目标。
我们如何知道目标的条件已得到满足？ 为了便于实施，我建议我们知道当用户到达特定 URL 路径时，目标已满足其条件。 作为额外的好处，我们可以在不弄脏视图的情况下测量这一点。 回到我们注册用户的示例，我们可以说当用户到达 URL 路径时就已经实现了这个目标：
/registration/complete
因此，我们定义
a_b_goal_conditions_met：路径
当考虑 Django 中的路径时，很自然地就会想到使用不同模板的想法。 是否还有其他方法还有待探讨。 在 A/B 测试中，您可以在两个页面之间产生微小差异并测量结果。 因此，最佳实践是定义一个基本路径模板，所有通往目标的路径都应从该模板扩展。
应该如何渲染这些模板？ 装饰器可能是一个好的开始 - Django 中的最佳实践是在视图中包含参数
template_name装饰器可以在运行时更改此参数。您可以看到这个装饰器要么内省包装的函数并修改
template_name参数，要么从某个地方（例如模型）查找正确的模板。 如果我们不想将装饰器添加到每个函数中，我们可以将其作为 ABMiddleware 的一部分来实现：我们还需要添加一些方法来跟踪哪些视图正在运行 A/B 测试等。
一种将观众沿着路径发送的系统
理论上这很容易，但有很多不同的实现，因此不清楚哪一个最好。 我们知道一个好的系统应该沿着路径均匀地划分用户 - 必须使用某种哈希方法 - 也许你可以使用 memcache 计数器的模数除以路径数 - 也许有更好的方法。
用于记录测试结果的系统
我们需要记录有多少用户走了哪条路径 - 当用户达到目标时我们还需要访问此信息（我们需要能够说出他们满足目标条件的路径） - 我们将使用某种模型来记录数据，并使用 Django 会话或 Cookie 来保存路径信息，直到用户满足目标条件。
结束语
我已经给出了很多在 Django 中实现 A/B 测试的伪代码 - 以上绝不是一个完整的解决方案，而是创建一个可重用的 A/B 测试框架的良好开端在姜戈.
作为参考，您可能需要查看 GitHub 上 Paul Mar 的七分钟 A/B - 这是上述内容的 ROR 版本！
更新
经过对Google Website Optimizer的进一步反思和调查，发现上述逻辑显然存在漏洞。 通过使用不同的模板来表示路径，您将破坏视图上的所有缓存（或者如果视图被缓存，它将始终提供相同的路径！）。 相反，我不会使用路径，而是窃取 GWO 术语并使用
组合的想法 - 这是模板更改的一个特定部分 - 例如，更改code> 站点的标签。
该解决方案将涉及将呈现为 JavaScript 的模板标签。 当页面加载到浏览器中时，JavaScript 会向您的服务器发出请求，该服务器会获取可能的组合之一。
这样您就可以在保留缓存的同时测试每个页面的多个组合！
更新
模板切换仍然有空间 - 比如说您引入一个全新的主页并想要针对旧主页测试它的性能 - 您仍然希望使用模板切换技术。 需要记住的是，您必须找到某种方法来在页面的 X 个缓存版本之间进行切换。 为此，您需要覆盖标准缓存中间件以查看它们是否是在请求的 URL 上运行的 A/B 测试。 然后它可以选择正确的缓存版本来显示！
更新
使用上述想法，我实现了一个用于基本 A/B 测试 Django 的可插入应用程序。 你可以从 Github 上获取它：
http://github.com/johnboxall/django-ab/树/主
It's useful to take a step back and abstract what A/B testing is trying to do before diving into the code. What exactly will we need to conduct a test?
With this in mind let's think about implementation.
The Goal
When we think about a Goal on the web usually we mean that a user reaches a certain page or that they complete a specific action, for example successfully registering as a user or getting to the checkout page.
In Django we could model that in a couple of ways - perhaps naively inside a view, calling a function whenever a Goal has been reached:
But that doesn't help because we'll have to add that code everywhere we need it - plus if we're using any pluggable apps we'd prefer not to edit their code to add our A/B test.
How can we introduce A/B Goals without directly editing view code? What about a Middleware?
That would allow us to track A/B Goals anywhere on the site.
How do we know that a Goal's conditions has been met? For ease of implementation I'll suggest that we know a Goal has had it's conditions met when a user reaches a specific URL path. As a bonus we can measure this without getting our hands dirty inside a view. To go back to our example of registering a user we could say that this goal has been met when the user reaches the URL path:
/registration/complete
So we define
a_b_goal_conditions_met:Paths
When thinking about Paths in Django it's natural to jump to the idea of using different templates. Whether there is another way remains to be explored. In A/B testing you make small differences between two pages and measure the results. Therefore it should be a best practice to define a single base Path template from which all Paths to the Goal should extend.
How should render these templates? A decorator is probably a good start - it's a best practice in Django to include a parameter
template_nameto your views a decorator could alter this parameter at runtime.You could see this decorator either introspecting the wrapped function and modifying the
template_nameargument or looking up the correct templates from somewhere (like a Model). If we didn't want to add the decorator to every function we could implement this as part of our ABMiddleware:We'd need also need to add some way to keep track of which views have A/B tests running etc.
A system for sending viewers down a Path
In theory this is easy but there are lot of different implementations so it's not clear which one is best. We know a good system should divide users evenly down the path - Some hash method must be used - Maybe you could use the modulus of memcache counter divided by the number of Paths - maybe there is a better way.
A system for recording the Results of the Test
We need to record how many users went down what Path - we'll also need access to this information when the user reaches the goal (we need to be able to say what Path they came down to met the Condition of the Goal) - we'll use some kind of Model(s) to record the data and either Django Sessions or Cookies to persist the Path information until the user meets the Goal condition.
Closing Thoughts
I've given a lot of pseudo code for implementing A/B testing in Django - the above is by no means a complete solution but a good start towards creating a reusable framework for A/B testing in Django.
For reference you may want to look at Paul Mar's Seven Minute A/Bs on GitHub - it's the ROR version of the above!
http://github.com/paulmars/seven_minute_abs/tree/master
Update
On further reflection and investigation of Google Website Optimizer it's apparent that there are gaping holes in the above logic. By using different templates to represent Paths you break all caching on the view (or if the view is cached it will always serve the same path!). Instead, of using Paths, I would instead steal GWO terminology and use the idea of
Combinations- that is one specific part of a template changing - for instance, changing the<h1>tag of a site.The solution would involve template tags which would render down to JavaScript. When the page is loaded in the browser the JavaScript makes a request to your server which fetches one of the possible Combinations.
This way you can test multiple combinations per page while preserving caching!
Update
There still is room for template switching - say for example you introduce an entirely new homepage and want to test it's performance against the old homepage - you'd still want to use the template switching technique. The thing to keep in mind is your going to have to figure out some way to switch between X number of cached versions of the page. To do this you'd need to override the standard cached middleware to see if their is a A/B test running on the requested URL. Then it could choose the correct cached version to show!!!
Update
Using the ideas described above I've implemented a pluggable app for basic A/B testing Django. You can get it off Github:
http://github.com/johnboxall/django-ab/tree/master
如果您像您建议的那样使用 GET 参数 (
?ui=2)，那么您根本不必接触 urls.py。 您的装饰器可以检查 request.GET['ui'] 并找到它需要的内容。为了避免对模板名称进行硬编码，也许您可以包装视图函数的返回值？ 您可以返回一个
(template_name, context)元组并让装饰器修改模板名称，而不是返回 render_to_response 的输出。 像这样的事情怎么样？ 警告：我还没有测试过这段代码这是一个非常基本的示例，但我希望它能传达这个想法。 您可以修改有关响应的其他一些内容，例如向模板上下文添加信息。 您可以使用这些上下文变量与您的站点分析集成，例如 Google Analytics。
作为奖励，如果您决定停止使用 GET 参数并转向基于 cookie 等的东西，您可以在将来重构此装饰器。
更新如果您已经编写了很多视图，并且您不想全部修改它们，您可以编写自己的
render_to_response版本。If you use the GET parameters like you suggsted (
?ui=2), then you shouldn't have to touch urls.py at all. Your decorator can inspectrequest.GET['ui']and find what it needs.To avoid hardcoding template names, maybe you could wrap the return value from the view function? Instead of returning the output of render_to_response, you could return a tuple of
(template_name, context)and let the decorator mangle the template name. How about something like this? WARNING: I haven't tested this codeThis is a really basic example, but I hope it gets the idea across. You could modify several other things about the response, such as adding information to the template context. You could use those context variables to integrate with your site analytics, like Google Analytics, for example.
As a bonus, you could refactor this decorator in the future if you decide to stop using GET parameters and move to something based on cookies, etc.
Update If you already have a lot of views written, and you don't want to modify them all, you could write your own version of
render_to_response.贾斯汀的回答是正确的......我建议你投票给那个人，因为他是第一个。 如果您有多个视图需要进行 A/B 调整，那么他的方法特别有用。
但请注意，如果您只有少量视图，则不需要装饰器或对 urls.py 进行更改。 如果您按原样保留 urls.py 文件...
...您可以使用 request.GET 来确定请求的视图变体：
如果您想避免对各个 A/B/C/etc 视图进行硬编码模板名称，只需让它们成为模板命名方案中的约定（贾斯汀的方法也建议这样做）：
Justin's response is right... I recommend you vote for that one, as he was first. His approach is particularly useful if you have multiple views that need this A/B adjustment.
Note, however, that you don't need a decorator, or alterations to urls.py, if you have just a handful of views. If you left your urls.py file as is...
... you can use request.GET to determine the view variant requested:
If you want to avoid hardcoding template names for the individual A/B/C/etc views, just make them a convention in your template naming scheme (as Justin's approach also recommends):
基于 Justin Voss 的代码：
使用代码的示例函数：
这里发生了什么：
1、传递的UI参数保存在session变量中
2.每次都会加载相同的模板，但是上下文变量{{ui}}存储UI id（您可以使用它来修改模板）
3. 如果用户进入没有 ?ui=xx 的页面，那么在索引 2 的情况下，他将被重定向到“?ui=33”，在索引 1 的情况下，UI 变量设置为 0。
我使用 3 从主页重定向到 Google网站优化器又使用适当的 ?ui 参数重定向回主页。
A code based on the one by Justin Voss:
example function using the code:
What happens here:
1. The passed UI parameter is stored in the session variable
2. The same template loads every time, but a context variable {{ui}} stores the UI id (you can use it to modify the template)
3. If user enters the page without ?ui=xx then in case of index2 he's redirected to '?ui=33', in case of index1 the UI variable is set to 0.
I use 3 to redirect from the main page to Google Website Optimizer which in turn redirects back to the main page with a proper ?ui parameter.
您还可以使用 Google Optimize 进行 A/B 测试。 为此，您必须将 Google Analytics 添加到您的网站，然后当您创建 Google Optimize 实验时，每个用户都会获得一个包含不同实验变体的 cookie（根据每个变体的权重）。 然后，您可以从 cookie 中提取变体并显示应用程序的各个版本。 您可以使用以下代码片段来提取变体：
You can also A/B test using Google Optimize. To do so you'll have to add Google Analytics to your site and then when you create a Google Optimize experiment each user will get a cookie with a different experiment variant (according to the weight for each variant). You can then extract the variant from the cookie and display various versions of your application. You can use the following snippet to extract the variant: