Technical Debt

The term Technical Debt was coined by Ward Cunningham of XP fame. Technical debt is a burden and I am sure many will have experienced the following (which is not comprehensive):

• ‘We’ll do it in the next release’ – that never comes.
• The project got delivered – along with a maintenance team.
• We did it for good business reasons.
• “The architecture - let me see - I have a single power-point somewhere with a picture on it”.
• “To get anything released to production is a nightmare - we have to regression test the whole system”.

The third argument is especially common when you are a young company and you have very little to lose by getting stuff into production - so you just go for it. But when you have thousands of customers the stakes are then much higher. It is no surprise that you hit a brick wall and end up in release hell.

An excellent article on Technical Debt by Kane Mar :

And another excellent article by Steve McConnell:

…

One approach to reducing technical debt is to have a technical debt sprint at the end of several business focused sprints. The argument that is put forward to the product owner who has responsibility for prioritizing each release is that reducing technical debt will increase your velocity on subsequent sprints more than the cost of refactoring during a TD Sprint.

Emergent Architecture - yes or no?

I have worked on some massive Programmes of work over the years. In one case the Architecture team for the programme must have been about 15 people, and the development community hit about 400 people. We waited for requirements, those requirements changed, we re-architected etc. The waste must have been huge. The cost certainly was. One ‘wag’ said ‘never in the history of IT has so much been produced by so many for so high a cost,only to have been re-worked’. What he forgot to say was that the supplier was on a time and materials basis and was winning all the way to the bank! We collectively dug a hole, and we got out of it by using techniques that are commonly called RAD - one of the roots of SCRUM (IMHO). The project went on to be a major success but it could have been so much easier. Lesson learned.

One argument in favor of emerging architecture employs industry statistics that show 35% of requirements change during the project - along with the other fact that over 50% of them are rarely or never used. You only have to open up Microsoft Office to figure out that statistic is probably true some of the time. So - if requirements change so much - why bother doing all that hard architecture work up-front! Be careful! the statistic being banded about seems to refer to large projects i.e 100K function points and above!

I have also worked with people who seem to be in a ‘features war’ with competitors, and are afraid that if we do not have this bell or that whistle that the customers will leave. Ask them to quantify it and you will be amused at some of the answers. So even more ‘evidence’ in favour of the argument. I have certainly seen projects that are a ‘must’, only to have been proved to be a ‘maybe’ in which case virtually all the requirements were a waste of time. Hmmm ..business case anybody?

The flaw in the argument is that these statistics are an average (for large projects), and your project may well be different e.g. a straight code conversion project would have a simple requirement - e.g. ’same functionality, in C++ and 10 times faster’ - and hardly a user in sight - since it is pretty much a non-functional requirement - even if it emanates from a department that might want something to be faster for good business reasons (faster = can process more with less = cheaper etc).

SCRUM - or at least Ken Schwaber is a proponent of emergent Architecture and Design. Even though SCRUM does not propose engineering practices this proposal does smell like an engineering practice to me

build at least one piece of business functionality every Sprint
build enough of the Architecture and Design to support that business functionality
build adequate architecture and design to meet the non-functional requirements

Then, every subsequent Sprint, more and more of the Architecture and design emerge, in response to more and more business functionality. A friend of mine refers to this as ‘make it up as you go along’ haha. However its not meant to be that, but it is certainly a hard sell - especially to architects (I used to be one so maybe I am schizophrenic on this!).

The purpose is to build sufficient architecture and design in response to actual, high priority business needs, to prove the business need can be satisfied by the Architecture. If the architecture team state that is not Optimal then I refer to Steve McConnell on this ‘Optimal for who? - the business or you?’.

However, if you do use this approach you need to be very careful indeed. The aim should be to ensure each Sprint ends with code that is clean, commented and refactored. If you do not do this then you are into delivering more technical debt which will become a burden over time. Spaghettii Architecture anyone?

Some methodologies e.g. DSDM propose prototyping prior to getting down and dirty with a full team. There are a lot of advantages to the prototype approach - you get to figure out what you want before you commit too much resource and then you build it - ’safe’ in the knowledge that you have ironed out most of the issues early on in the process. I must admit to preferring this approach. Then I get clobbered with - ‘A SCRUM sprint is like a prototype but with ’shippable code quality’ at the end of it’.Heres a link to Sutherlands Blog which also discusses this:

Scott Ambler has written an interesting Blog item on ‘Agile Architecture’ and the approach. I believe it is a more ‘workable’ and considered approach to Agile Architecture than the more XP type views.

In the book by Stephens and Rosenberg ‘Extreme Programming Re-factored: The case against XP’ there is a whole chapter devoted to the debate around Emergent Architecture and Design. They address the fear of Big Design Up Front (BDUF) head on and state that the customer can get to see visible progress early on and easily:

1. Break the design into iterations, produce a high level design (architecture) that can then be divided into sub-systems, each of which is then designed in more detail. The Subsystem may be designed and implemented concurrently, or sequentially in separate iterations.
2. Showing the customer the early prototype. Remember the ‘prototype’ may be many disparate pieces of code, each intended to explore a different area of the design.

The authors argue that Emergent Design is useful when used in moderation and under the correct circumstances. They also argue however that XP appears to be stuck in the initial “on the face of it” reaction to design - almost echoing my friends comments of ‘make it up as you go along’ or - “jump in and repent at your leisure”. The authors also argue that it is possible to get a design close too correct - i.e fairly stable - if you use the right approach recognizing that it will then emerge. For what its worth, I tend to agree and have been in the fortunate position of having done it. I think the argument for Emergent Architecture could win fans if it simply was not so extreme!