Archives for November 2009

Test-driven Development in Agile Projects

Profile picture for Giv ParvanehGiv Parvaneh

Post categories:

15:15 UK time, Thursday, 19 November 2009

Comments (6)

Hi, I'm Giv Parvaneh, and I'm a senior PHP web developer at the BBC. When I started working here earlier this year, I quickly realised that I would have to re-think the way I develop web applications. As a public service and one of the most visited websites in the UK, the BBC has an obligation to deliver quality products; the websites we build not only have to conform to web standards and be fully accessible, they need to be scalable to handle millions of users. Quickly hacking things together isn't really an option here...

The need for speed

Developers at the BBC tend to use Agile methodologies as a way to quickly release iterations of products. But where does rigorous code testing fit in with the short development and release cycles? How can we maintain the quality of our code when things need to change so fast?

A confession

OK: I'm guilty of releasing untested code in the past, where I've needed to meet deadlines. I'm probably not alone. And if you're a developer who's worked in a big team, you'll know that often you'll have to work with code that was written by your predecessors or contractors who didn't create 'test-friendly' code. Often this means having to waste time re-writing the code so that it can be tested. Naturally, this eats into your valuable development time during sprint cycles.

The Agile Manifesto states that we should value "Working software over comprehensive documentation". Some might see a problem here, because this seems to place a higher importance on "working software" than the quality of that software. The way I see it, you can't truly maintain "working software" if it's not well-tested... and the truth remains that many developers compromise good test coverage in order to complete tasks by the deadline. So what can we do?

A compromise

One thing I love about Python is its built-in support for documentation and testing. Even if you're not in the habit of writing unit tests for your code, you should at least comment your code so that you and other developers know what your classes and methods do. Python's "doctest" kills two birds with one stone: as you write your comments, you can optionally specify how the code should be used and what the expected outcome should be. Comments now serve as valuable documentation for your code, and can be executed as a test to ensure your logic is intact.

This method is not supposed to replace unit tests. In fact, there's only so much you can do with doctests, and your code can quickly become a huge mess if you try to write too many. But the idea is that they can be quickly used when you don't have time to write a full test suite. Some tests is better than no tests.

Let's have a look at an example in Python first:

def greetings(your_name): return 'hello, %s!' % (your_name)

This function takes a person's name as an argument and returns a greetings message. Let's add this description and a test to the docs:

def greetings(your_name): """ This function takes a person's name and returns a greetings message >>> greetings('Auntie') 'hello, Auntie!' """ return 'hello, %s!' % (your_name)

When you execute this script, Python is smart enough to know ">>>" means the comment needs to be interpreted as code and what the returned value should be.

We use PHP at the BBC, so I did some research and came across a PHP equivalent called PHPDT, which can be installed via Pear. Here's the PHP version of the above function:

/**
* This function takes a person's name and * returns a greetings message
* 
* greetings('Auntie');
* // expects:
* // 'hello, Auntie!'
* 
*/
function greetings($your_name)
{ return 'hello, ' . $your_name . '!';
}

You can test this by running $ phpdt example.php in your terminal. The comments inside the <code> tags will be interpreted as PHP code.

Testing times

These are simple examples, but you can see how easily you could combine your comments with some testing to make sure that any accidental changes in your code are caught. Again, doctests aren't meant to replace unit tests, but they might help to keep things working when you're faced with the need to write code faster than your tests can keep up.

I plan to try adding doctests to my code, as well as continue writing proper unit tests in PHPUnit. I'd be really interested to hear from anyone who has tried this approach: on large applications, is this a good way to move fast without letting your test coverage slip?

Extending OpenSocial and Shindig: a crash course

Profile picture for Ben SmithBen Smith

Post categories:

11:20 UK time, Friday, 6 November 2009

Comments (5)

Hi, I'm Ben Smith, and I’m developing a new set of web services that add social features to BBC Online. Starting this project was clearly an exciting prospect, but without a handy crash course on how to get going, we ran into our fair share of problems. Now that we’ve solved a few of them, I thought I’d better start writing about it.

Forging ahead

The BBC’s overall web architecture has been rebuilt over the last couple of years, and a new breed of websites is now starting to appear (the platform's called 'The Forge', and you'll hear more about it soon). In a nutshell, these new sites are delivered through an application layer that relies on making HTTP calls to RESTish web services to get the data they need.

If we want to develop something social, we need some social web-services, and you don’t have to look too far into this before the OpenSocialRESTful specification jumps out and does a little dance in front of you. (Well, that’s what it did for me, but I may have been drinking too much coffee.)

It’s Open and it’s Social

OpenSocial defines a common set of APIs that applications can be developed against. The REST API allows applications to make HTTP requests, and get back things like a list of a user’s friends

/people/<userId>/@friends

or their recent activity

/activities/<userId>/@friends

Lovely stuff. Not only do we have a specification to work to, we have a popular one that many applications have already been developed against.

Additionally, to aid in the development of OpenSocial-compliant sites, the Apache Incubator project Shindig provides OpenSocial implementations in both Java and PHP. As the BBC’s new platform is particularly fond of developing web services in Java, it’s all looking pretty good.

Is that it?

Unfortunately, life is never that easy. OpenSocial is designed to be a generic API to present your data through, whether you’re a Google, a MySpace or a jonnynewsite.com. But different websites have different needs: different relationship models, and various data and business logic around the creation of things like new users or relationships between them. It would be presumptuous of OpenSocial to define how a website should approach them, so it doesn’t.

For example, while OpenSocial allows clients to get a user’s information:

/people/<userId>/@self

it doesn’t define a way to create a new user. So we have to do these things ourselves.

A bit of a Shindig

Shindig, the OpenSocial container implementation, quite rightly implements the OpenSocial specification and no more. As everyone will have their own infrastructure, with different databases, key-value stores and identity systems, Shindig defines software-level interfaces for you to implement. So, for Shindig to be able to respond to:

/people/<userId>/@self

you need to implement:

Future<Person> getPerson( UserId id, Set<String> fields, SecurityToken token) throws ProtocolException;

which is defined in the PersonService interface.

Extending OpenSocial

The BBC website has no social graph or any real social features to speak of. This meant that when we started, we were presented with an interesting opportunity: with no existing system to integrate with, could we extend the OpenSocial spec, and Shindig itself, to provide the features we were missing?

As I mentioned before, OpenSocial allows clients to retrieve a user’s friends:

/people/<userId>/@friends

So, to create a friendship between two users that’s consistent with the existing API calls, it seemed reasonable to POST a document (JSON in this case) that contains the new friend’s userId to the same URL:

POST {id: ‘<friendId>’}
/people/<userId>/@friends

Some may argue that a PUT to:

/people/<userId>/@friends/<friendId>

would smell more RESTful, but it’s not really any better and, considering the criticisms of OpenSocial’s RESTful-ness, keeping the API consistent was most important. However, if we were, say, keeping a description of the type of friendship (colleague, family, etc.) then we would need to be able to update that metadata, and would have PUT to the specific friendship resource:

PUT {type: ‘colleague’}
/people/<userId>/@friends/<friendId>

Extending Shindig

To add this function to Shindig, it’s a matter of extending the right bits. I’ve created a repository on github.com that shows how you can do this, but it’s worth saying that this isn’t production code, especially as it is backed by a non-persistent JSON store (and probably has more bugs in it than a student’s beard). As I mentioned earlier, Shindig specifies a service layer for you to implement. Conveniently, they also provide a sample implementation that persists to an in-memory JSON store which is populated with dummy data.

Shindig is sensibly split into layers: the domain model layer, the service layer, and the request handling layer (there’s actually a lot more to it than that, but you can read about that here). To extend the RESTful API you have to define the new endpoint in the appropriate handler, and a service that will administer the appropriate model objects.

So, to allow clients to POST new friendships to /people/<userId>/@friends, you must firstly extend the PersonHandler:

@Service(name = “people”, path = “/{userId}+/{groupId}/{personId}+”) public class PersonHandlerImpl extends PersonHandler {

Now you can implement a createFriends() method that will respond to POSTs to /people/<userId>/@friends:

@Operation(httpMethods = “POST”, path = “/{userId}+/@friends”)
public Future<?> createFriends(SocialRequestItem request) throws ProtocolException {

You can see exactly how this is done in PersonHandlerImpl. The annotations are Shindig’s own home-brewed routing mechanism, but they’re pretty standard MVC controller fare. The handler itself shouldn’t do the work of creating a friendship, but should leave this to a service interface for the management of relationships:

public interface RelationshipService { public void createRelationship(String personId, String friendId);

Now you have to implement the RelationshipService, in much the same way as you need to implement Shindig’s own standard services. As this is an example I simply extended the JSON implementation to include a fleshed out createRelationship() method:

public class JsonDbServiceExample extends JsonDbOpensocialService implements RelationshipService { public void createRelationship(String personId, String friendId) {

Dependency issues

This is all fine and dandy, but how do any of these pieces actually know to use each other?

Shindig uses Google-Guice, a lightweight dependancy injection framework, which you use to tell the system that you actually want to use your handler and service implementations over theirs. To do this, you simply reference your own GuiceModule in your web.xml or tests.

It’s also worth mentioning that you can integrate Shindig and Guice with Spring, which also provides dependancy injection and is very popular, as Chico Charlesworthdocuments on the Shindig wiki.

Anyway, if you fancy trying it out, gitclone the example and, assuming the code still works, tinker to your heart’s content.

BBC © 2014The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.