Gathering and Analyzing Data with Django: Part 2

Instapaper Text

Gathering and Analyzing Data with Django: Part 2

After a bit of a delay here’s part 2 of Gathering and Analyzing Data with Django. The previous post can be found here, which has had some minor changes since posting. The goal of the series is to showcase Django as a way to build web-enabled tools. The example presented is an analytics site for gameplay data. In the previous post we setup the site and created the database model. In this post we’ll begin collecting the logs and populating the database.

Gathering the Game Data

With our models in place we can now implement a way to gather the data. The most flexible way to do this is to have our game create a log file detailing the player’s experience. At the end of a session this can be uploaded to the analytics site. From there it is parsed with its information compiled within the database.

The first piece of the puzzle to tackle is parsing the log file. To keep things simple we’ll generate a comma-separated values file. A library to parse this brand of file is already present in Python.

 
  08:00:00,QUEST_STATUS,FET0001 ACCEPTED
 
  08:30:00,QUEST_STATUS,FET0001 COMPLETED
 
  08:42:00,QUEST_STATUS,FET0002 DECLINED

Create a utils directory containing log.py, quest.py, and __init__.py. This will keep our application organized when additional events require parsing.

The parse_log function handles reading the file. It simply reads a line from the file, looks at the event, and forwards it to the corresponding function for additional processing. To make things more scalable in the future we’ll use a dictionary of function pointers to specify what handles the message.

import csv
 
  from datetime import timedelta
 
  from quest import log_quest_progress
 
   
 
  log_functions = {
 
  	'QUEST_STATUS': log_quest_progress,
 
  }
 
   
 
  def parse_log(player_id, log_file):
 
  	print player_id
 
  	log_reader = csv.reader(log_file)
 
   
 
  	for row in log_reader:
 
  		if len(row) == 3:
 
  			hrs, min, sec = map(int, row[0].split(':'))
 
  			time = timedelta(hours=hrs, minutes=min, seconds=sec)
 
   
 
  			event = row[1]
 
   
 
  			if log_functions.has_key(event):
 
  				func = log_functions[event]
 
  				message = row[2]
 
  				func(time, player_id, message)

The update_quest function constructs and updates the player’s progress through the associated quest. It has a bit more to do as it parses the message text into the required database fields.

import re
 
  import logging
 
  from django.db import models
 
  from game.models import *
 
   
 
  quest_regex = re.compile('(?P<code>\w+)\s+(?PACCEPTED|DECLINED|COMPLETED)')
 
   
 
  logger = logging.getLogger(__name__)
 
   
 
  def log_quest_progress(time, player_id, message):
 
   
 
  	m = quest_regex.match(message)
 
   
 
  	if m:
 
  		quest_code = m.group('code')
 
  		print quest_code
 
   
 
  		try:
 
  			player = Player.objects.get(id=player_id)
 
  			quest = Quest.objects.get(code=quest_code)
 
   
 
  			quest_state, created = QuestState.objects.get_or_create(player=player, quest=quest)
 
   
 
  			status = m.group('status')
 
   
 
  			if (status == 'DECLINED'):
 
  				quest_state.status = QuestState.DECLINED_STATUS
 
  			elif (status == 'ACCEPTED'):
 
  				quest_state.status = QuestState.ACCEPTED_STATUS
 
  				quest_state.start_time = time.total_seconds()
 
  			else:
 
  				quest_state.status = QuestState.COMPLETED_STATUS
 
  				quest_state.end_time = time.total_seconds()
 
   
 
  			quest_state.save()
 
   
 
  		except Player.DoesNotExist as e:
 
  			logger.error("Player does not exist")
 
  		except Quest.DoesNotExist as e:
 
  			logger.error("Quest does not exist")
 
  	else:
 
  		logger.error("Invalid message")

Working with Forms

With our log parsing function in place the next step is constructing the form to upload to. To handle the processing of posted data Django provides a Form class. As with models, forms simply descend from the Form class and specify their fields. By convention any forms the application uses are placed in forms.py.

To use the form a view needs to be specified. Open up views.py and add the upload_form function.

from django.shortcuts import render_to_response
 
  from django.http import HttpResponse, HttpResponseBadRequest
 
  from forms import LogForm
 
  from utils.log import parse_log
 
   
 
  def upload_log(request):
 
  	if request.method == 'POST':
 
  		form = LogForm(request.POST, request.FILES)
 
  		if form.is_valid():
 
  			player_id = form.cleaned_data['player']
 
  			log_file = request.FILES['file']
 
   
 
  			parse_log(player_id, log_file)
 
   
 
  			return HttpResponse()
 
  		else:
 
  			return HttpResponseBadRequest()
 
   
 
  	return render_to_response('upload_log.html', {'form': LogForm()})

A view always receives a HttpRequest, which has a property specifying its HTTP method. One of the properties specified is the HTTP method. By looking at this we can determine whether the form has been completed. If its complete the next thing to check is its validity. The Form class has this functionality built into it so we can just let it do its thing. From there we pass the file on to our log function.

If there is an error with processing we need to notify the client of this. Since the log file is being uploaded programatically we can convey this message by using a status code. HTTP defines a number of status codes. In our case we send a HttpResponseBadRequest which sends a 400 status, meaning the request cannot be fulfilled due to bad syntax. The client can then reattempt the upload at a later time.

To finish off we need to create the template file specified in the view. Add a folder to the root of the project with the name templates. Within it create upload_log.html.

<html>
 
  <head>
 
  	<title>Upload Log</title>
 
  </head>
 
  <body>
 
  	<form action="" method="post">
 
  		<table>
 
  			{{ form.as_table }}
 
  		</table>
 
  		<input type="submit" value="Submit" />
 
  	</form>
 
  </body>
 
  </html>

The file itself contains the bare minimum. The form is passed as a parameter from the view during the processing of the template. The Form class comes prepackaged with a method to generate the appropriate HTML. While this page won’t win any design awards it is sufficient for our purposes.

Before we can view the form we need to add our template directory to our configuration. This is specified in settings.py in the TEMPLATE_DIRS variable.

TEMPLATE_DIRS = (
 
  	'./templates/'
 
  )

Since we’re adding a view that handles post data an additional middleware should be installed. This is the CsrfResponseMiddleware which is a security method against cross-site request forgeries.

The last step is to route a URL request to the view. This is done by modifying the urlpatterns within urls.py.

urlpatterns = patterns('',
 
  	('^upload_log/$', 'game.views.upload_log'),
 
  	('^quests/', 'analytics.views.quests'),
 
  	(r'^admin/', include(admin.site.urls)),
 
  )

Finally we can access our form by going to http://127.0.0.1:8000/upload_log/.

Upload Form

Up Next

In the next installment we’ll take the data gathered and prepare it to be visualized. We’ll be going over jQuery.

#AltDevBlogADay

Don Olmstead
Follow @don_olmstead