Using new practices to promote agile culture

August 14, 2009 in Development, Lean, Methodology, Teamworking

As I’ve been saying, we need to change a few things around and some oganisation  things are already happening.

But that will only work if the people involved change their daily habits and mentality. It is perfectly possible to work within a RUP organisation and still maintain silos of knowledge where each developer is an expert in an area and work stops if he or she is not around. And I am yet to see a projet methodology that ensures real quality, rather than documenatation indicating it. It is easy writing up a long test script and showing the results, but does that ensure the quality of the application? Andwhat about the architectural scalability, and maintainability?

These things can only be addressed by changing the way the people involved in the project work and think on a daily basis. The quality we are after only comes from people caring about what they do and catching any potential issues early.

So we’ve started to use some practices that have been around for a while.

• Test driven development for all new development, and legecy apps where it is possible. We have an architecture that is stopping us though, it is about 5–7 years old and using EJB’s that are heaviliy reliant on the application servers EJB container, and we dont have a test harness.
• Continuous integration. At the moment, since we have that ancient EJB solution  this only makes sure the compilation does not fail. But its a first step.
• Code reviews, from now on we’ll be peer reviewing any new release, both to make sure we get a second pair of eyes on it to validate the solution (we’re still not pair programming) but also to spread the knowledge.
• Quality metrics. A touchy subject, but if we can clean up the code and use checkstyle, code coverage, static code analysis, we can get rid of some low-level checks we’re doing manually, and also get warnigns if the code base is losing maintainability.
• Automating functional tests. At the moment all testing is manual, but with a little effort, each manual test can be automated and in time a full suite can be developed and our testers can do some real testing.

We’ve also started some discussion forums and a more formal quarterly meeting where we can discuss what w can do better to ensure we’re not staying static.  Since we have a relatively low staff turnover, there are  few ideas coming in from other industries and companies so in order to evolve we need to continuously asses, evaluate and adapt. And read a lot of blogs to find out what others are doing, of course…

So hopefully, these efforts will lead to increased communication on the ‘shop floor’ and an increased attention to quality and detail. And once the codebase has gotten a bit better, some knowledge is shared and with some automation in place, things can start turning around a bit quicker.

But most importantly, we can stop wasting time on activities that are non value adding and start spending time commuinicating instead. By using tools and practices sensibly to assess we are building things right we can start building the right things.

And thats nothing a process can help us with.

Introducing an Agile Development process

July 20, 2009 in Development, Teamworking

Currently, we’re not really running projects to implement functionality. Its more like small requests which are usually undertaken by an analyst, a single developer and a single tester, loosly connected, but with very specific roles and responsibilities.

• The analyst works with the business owner to formalize a specification and is responsible for the completion of the work on time and within budget.
• The developer is responsible for implementing the functionality and ensuring that the release of the patch (usually we are working with maintaining existing applications) goes smoothly.
• The tester is responsible for ensuring that the functionality described in the specification.

This is not an uncommon division of labour and there is no reason why this should not work however we are experiencing some problems.

1. The requirements are usually completed without much involvement from developers or testers resulting in specifications that are incomplete and sometimes impossible or undesirable to implement.
2. Since we are using a waterfall approach any problems with the requirements are discovered inside the development phase .
3. Once the development starts the tester is usually disconnected and any subsequent changes that occur are not communicated and therefore not reflected in the test plan.
4. Documentation is not coherent – each project may or may not provide documentation about the functionality and overall architectural documentation is scarce and out of date.
5. Poor support for new joiners since the process and the division of responsibilities between the roles are not documented

This all leads to a lot of projects being delayed in various parts of the process, because they have to jump back up the waterfall, and that leads to our biggest problem. : There are too many projects going on at at the same time, with people working on multiple tasks simultaneously, and this is bad!!! Not only are there overheads for each person involved, but different projects have different staff so people are on several teams at the same time. That will lead to resource contention and prioritisation failure since no one is able to see the wider picture and therefore everyone optimises their own workload, even though that might lead to the entire chain (all projects) being less efficient (p 38 onwards of this book describes just that).

What this leads to is a lot of incomplete work that will have to be pulled last minute, a lot of rework because of poor communication, wasted efforts and inconsistencies – both in functionality and quality.

New methodology

To solve all these problems we have been working on introducing a project management methodology that will allow us to address these problems.

• Flexible enough to apply for several different sizes of projects
• Well established in the industry to ensure that new joiners will be able to grasp it
• Enforce communication
• Focus on Iterative and incremental development

How do we turn this agile?

Now, my goal is hand has always been to turn this agile, and to be completely honest we do have some practices and procedures in place that can be used, but there are ‘buts’

• We have a backlog of ‘features’ (or various sizes of projects) , -but they are in different queues.
• We are developing in a feature driven way , – but these ‘features’ or projects can be decomposed further
• We already set up a team and project organisation around a ‘feature’ , – but teamworking is an issue
• We have scheduled monthly releases and release what is ready when the date arrives , – but we frequently fail to meet the release date
• We use source control and version management in this manner already , – but we only just started

Now this we can work with…

As a starting point, we have chosen to go with RUP, because

• it comes with a nice set of role and responsibilities descriptions,
• it has a large repository of predefined processes
• it is much less contentious than ‘Agile’ and is likely to be approved politically
• it can be scaled down enough to become Agile, and still please management in having a ‘phased approach’
• there is some prior experience within the organisation, which will help introduction and learning
• it will force us to document our systems in a coherent manner.
• the focus on ‘team’ is strong, it will allow us to build a cross-functional feature team.

We’re basically thinking of the Dreyfus model here. RUP will be able to give ample support to take us through the Novice and Beginner phases, after that phase when we have learnt enough we can start chopping away the fat. And there will be a lot of fat to chop. In the initial incarnation the project manager and the architect will be inundated with documentation, so the first couple of projects will be slow and painful. But with each project we’ll start building up our repository of documetnation and the next project will get it slightly easier and we’ll soon have a decent view of our applications.

Rolling it out will start with with general presentations, so the entire organisation is aware of what is happening, and in agile talk, making us all pigs rather than chickens.
Once that is done, the nexts step is workshops for those involved in the early projects in order to get familiarised with the process. Hopefully this will reduce the friction within these projects and people would like to work in another RUP project once the first one finishes.
But these projects will also need a lot of guidance, hopefully we’ll get a mentor in to guide us through the first couple of iterations wh can steer us in the correct direction

This will no doubt require a lot of discipline and support by management. Hopefully, by setting the expectations from the start and making it clear what the benefit is will aid that.

Introducing lean by stealth

May 27, 2009 in Architecture, Development, Methodology, Teamworking

I am currently finding myself in a team that I find is not reaching its potential or achieving even close to the results that I would expect from such a large team, with the experience available and the marketplace we are operating in. In particular I think that the interaction with other teams and departments, or lack thereof, is causing a lot of wasted time and pain. I have previously been working in agile teams and I’ve been reading a lot of material lately on agile methodologies and lean. I am trying to gradually steer our organisation towards being ‘flexible’, I don’t think there be a big bang approach to these things and I definitely think there will be strong resistance to such an initiative.

A brief background

The development team in which I joined a little while back has no real process, sure – one exists but it is incredibly convoluted and I am yet to find a development project that uses it.  What this ultimately leads to is that each project that is undertaken reverts to the informal, unstated process, or ‘the way things have always been done’.  The result of that is, perhaps as expected,  that the same mistakes are repeated and the improvement opportunities are being overlooked. During the last couple of years there has been some exposure to different methodologies, though these have been less than successful and inspirational.

The first one was a supplier who was pushing Scrum  who were involved in a project that eventually failed. Their failure to deliver sufficient documentation or indeed working code that matched the SLA was attributed to the methodology, rather than poor execution of it.

In more recent time a partner was involved in a long running implementation and with their experience in the  subject area came their preferred implementation methodology. The project did deliver functionality, though not all of it, and on time with a reasonable amount of issues for its size and as such was a success. However most people involved were not overworked and frustrated at the end and would not use the methodology again given a choice.

These two deterrents, combined with internal politics and the fact that turnover is rather low means that there is an innate resistance to change. This due to the fact that there is a culturally strong core that are comfortable with the informal process and protect it, while new influences are finding it difficult to get a foothold. I’ve heard this referred to as a hub and spoke model, an analogy that I find quite fitting with the hub concentrated, strong and central and spokes often unconnected and weaker.

The way forward

Turning this organization towards lean sounds insane, and in fairness it probably is but any movement, however small, in that direction will be beneficial.  I remember a conversation with a former project manager, and good friend, of mine a few years back when he was trying to do roughly the same thing. At that time we decided to shield ourselves, and create an ‘interface’ to the external stakeholders, and internalising those roles. While this approach worked rather well I feel is not adequate in this environment. Firstly we are interfacing with too many stakeholders and secondly bringing the teams closer and improving communication is imperative.

I think we are roughly talking about four major areas of change…

• Introducing Project management. We have not really had any real project management on any scale, the lead developer has been taking on a lot of responsibility. Introducing a project management methodology that is well known and recognised by a large part of the developmet community which offers enough flexibility that we can incorporate agile or lean concepts. The methodology also has to bring the different parts of the organisation closer together.
• Agile (Enterprie) Architecture: Another concept that has not been prominent thus far. The numerous application (in-house and off the shelf) are not working together and the continuous evolution of the architecture is not handled in an optimal way. Introducing an Enterprise architecture framework that allows for architectural agility and works well with the project management methodology is imperative. After all, for a software development organisation the architecture paves the way.
• Introducing new practices: From the bottom up there are several problems in the development team itself which hinder agility.  Historically a lot of the work has been done in isolation with lmited knowledge sharing and communication coupled with a lack of attentions to principles like SOLID and DRY. For a sporting chance of successfully reaching anything resembling agile there are several practices that needs to be put in place.
• Adapting the culture: Having frameworks and methodologies will get you a long way but the core underlying culture needs to embrace the nature of agility. The reluctance to leave the safety zone and try new ideas is a major hindrance, creating success stories in the small increments that I plan to bring these changes in are imperative.

I realize this probably has to be a gradual approach in several organizational layers at the same time, which is not making it any easier. The management need to adopt the project management and enterprise architecture efforts. Other teams need to buy into the project management methodology and the development team needs to work closer with these teams and change the way we work internally.

Daunting.

Fortunately there is a good deal of support throughout the organisation and I have no doubt that together we’ll be able to turn this thing around. At least its worth a try.

The Corporate IT politics game

March 23, 2009 in Development, Teamworking

In my rss reader there is usually a great deal of wisdom appearing each morning, and today was no difference. But its not often you get one of your problems, described in one and have it partially answered in another.

David Christiansen, author of the superb Technology Darkside blog tells this story, and I can see great similarities with corporate IT. In this case David talks about a dysfunctional team of consultants, but isn’t the same game played everyday in corporate IT? I think its called politics. So take those points and add

• getting that senior-, specialist-, expert-, whatever added to your title,
• creating endless cc lists on your email to prove you’re working hard and at the same time provide ass-coverage.
• coming into the office while nearly-dead, since sick-days are just not for winners and of course
• the hero anti-pattern.

All this, and many more, in order to keep climbing that corporate ladder. It takes a good manager to recognise these and the fact that they are counter productive and to take actions against it. One very effective cure to these things is to create total transparency to promote recognition by merit. The way Jurgen Appelo describes it here makes perfect and powerful point. One of my previous project managers didn’t actually post the salary spreadsheet on the wall but he threatened to and the effects were quite interesting.

But while this is very contentious, there is plenty of information that is not so and there are lots of tools out there today that makes information about your projects and product readily available. Why not let your continuous integration tool (you do have one, right?) generate a project report each night and publish it to the intranet. And while we are talking about that nightly build, it will break and when it does – make sure its very public, and slightly embarrassing. Project reports could also be public knowledge, which project managers are continuously over budget, which developers keep underestimating their efforts, which testers can’t find bugs? But more important than showing shortcomings, achievement can publicly proclaimed and rewarded. When the builds haven’t failed for a week, its the team leads responsibility to buy the first round of beers, or get that Friday sugar high going  creating that sense of collective achievement.

After all, until this information is in the public domain, how are you going promote recognition by merit? Lets  hope more
corporate IT departments are opening up to Agile and Lean, where
openness and teamwork is the norm.

//J