How to Justify Scrum to Senior Management

Often many people say that it is difficult to justify Scrum to the senior management. I think there are two alternative approaches you can take to justify Scrum:
If the senior management is willing to listen , you can tell them about the following scrum benefits:

• None of the software development methodologies can predict the exact date of software completion with exact amount of resources. In most cases where inputs and outputs are not well defined, empirical process control model is the only workable model for software development. Read chapter 5, Agile Software Development with Scrum.

• Prioritized product backlog ensures that 80% of the business value is delivered in 20% of the time. Read Appendix D, Agile Project Management With Scrum.

• Product gets a legitimate chance to change their product based on sprint demos. There is no need to file a change request and charge more in case of a fixed bid and/or fixed date contracts. Scrum being a transparent process, it gives more visibility to the management than waterfall, which provides no visibility till the time the product is finally released.
  

• Once Scrum is in place, it is possible to estimate release dates roughly based on historical velocity data. In fact, you will be able to predict release dates of the current project with 80% probability after 20% of the work is done - within 3 to 5 sprints based on the project size.

• Scrum exposes problems earlier, there by giving a chance for management to cancel the project if necessary.

• If you are a high maturity company, then using Scrum will improve productivity and performance significantly as concluded by Jeff Sutherlands's paper - Scrum and CMMI Level 5: The Magic Potion for Code Warriors
  

If the senior management is not willing to listen, you have no choice but to do the analysis first and give them a date. You may not be able to score with a management not willing to listen to new ideas.

Point Score and Work Remaining in hours

A day before sprint 1 was about to end, the scrum master, team manager and scrum-trained developer were sitting in a meeting room trying to figure out the co-relation between velocity and estimated hours. The question in front of them was - What is the guideline for the team to choose stories? Should we give them a velocity number or the team should make a decision based on total number of available developer hours?

If we are going to use the total number of available hours for picking up stories from the backlog then what is the purpose of velocity? Why do we estimate stories in points? Why use different units? Why not just use hours so that this whole estimation is uniform in hours ?

They had only one sprint worth of data. The team was new to Scrum. The manager felt strongly that they should give some guideline to the team so that the team does not overburden itself or does not take up little work. He also wanted to know the correlation between points and hours.

Here is what Ken Schwaber says in his book - Agile Software Development with Scrum "This (in our case the questions raised above) question indicates the difficulty shifting from a pert chart-based, time reporting structure to an empirical approach"

There is a difference between assigning a point score for a story and estimating hours for a story. Point score is the effort. Work remaining is the time (hours). The point score for a story is an approximate score. This score is used in velocity calculations. Product owner and the scrum master use this for release estimation and planning.

The team only uses estimated hours while picking stories for a sprint. They pick a story from the backlog and estimate the number of hours after clarifying requirements with the product owner. Estimating hours require more understanding of the story than assigning a very rough point score. Furthermore, estimation should be preferably done for tasks - at a finer granularity than story. A story should be divided into multiple tasks such as testing, coding, writing unit tests, coding UI layer etc. That makes a story more predictable. The team only deals in number of hours and specifically remaining number of hours. On realizing this, the manager stopped deriving the correlation between effort and time.

Why Agile is hard?

I was talking with some developers and a manager of a company in LA about their Scrum implementation. They told me about various problems they faced - no buy in from business, inability to estimate exact dates etc.

Interestingly, at the same time, I happened to listen to Why Agile is Hard - The java Posee round up 08 podcast. Same problems were discussed in the podcast. The podcast mentions many interesting books and websites for agile followers. If you are an agile devotee, you should definitely listen to this podcast.

The developers from the company also talked about implementing Scrum only in the engineering team without getting buy in from the business. The business will give them a document of requirements which will be translated into user stories by a team lead.

One developer even told me about doing database design out of scrum sprints to avoid future changes. I think this is fundamentally wrong and against the spirit of Scrum. This is a typical case when unit tests are not implemented. Code without unit tests is difficult to re-factor hence developers tend to think about future requirements; there by defeating the purpose of iterative and incremental approach

Their other problem was accommodating QA in a sprint mainly because the testing was manual. I tried convincing them QAing within the sprint instead of doing staggered QA sprints. I suggested that they should try to automate the tests. While developers are working on the code, QA should work on creating / changing the automated tests.

They hadn't read the bible - Agile Software Development with SCRUM (Series in Agile Software Development). I suggested them to start with the book. Careful reading of this book will answer many of their questions. I also agree with one of the person in the podcast - start with small changes. Specifically for people who don't have unit tests. They need to start with unit testing first.

Real Scrum, Pseudo Scrum

Before we embarked on the current, real scrum project, we did a project with pseudo scrum.

A big project was on the way. An enthusiastic engineer and his manager thought of giving scrum a try. They approached the company's project management department and the like minded project manager picked up the idea. None of these were trained in Scrum. They started reading the bible - Agile Software Development with Scrum.

The three musketeers got buy in from some more people and the project began. While the project was in progress, they fumbled, they sinned. Their daily scrums went on for 45 minutes!. They did not track remaining hours and there was no burndown chart. Developers flaunted their individual achievements in the sprint demos. They could not fit QA in the scrum and had to resort to parallel staggered QA sprints.

But they did not let the scrum spirit die. They religiously followed the incremental approach, their design decisions were based on the team consensus. The stories were end to end. They wrote lots of unit tests. The incremental, iterative approach caught on momentum. The engineering enjoyed it, the product rejoiced it. Their good karma paid off and project was a huge success! QA was amazed with the code quality.

They were able to build a new website with a complex domain model in mere 4 months. The whole company was amazed with this success. The success got them thinking about scrum seriously.

Few months passed by. A new big project was about to come. The project management team, realizing power of the scrum, decided to get Scrum Master Certification training. The training opened their eyes. They realized what they did wrong, as well as their good deeds. The training was just about right for the people who have tried scrum once.

This is how the current project began. It is our first attempt to implement text book scrum - the real scrum. I will keep you posted about our experiences through this blog.

I strongly recommend - just try it out! Once you try it, you will realize why it works

Company Policies and The Team's Design Choices

Our company is becoming more and more corporate day by day. Ironically as it becomes more corporate, it tries to adopt more agile methodologies.

One of the company's director is an expert in databases. Some teams in the company regarded him as a data architect. Eventually he was officially appointed as the data architect few months ago. We were asked to consult with him for any database schema design or other database related issues.

The data architect then took it upon himself to standardize solutions across various teams. He came up with a database schema design for a feature and proclaimed that every similar feature should use the same schema. Everybody, albeit reluctantly, agreed with his schema design. Now when a similar feature needs to built as part of the sprint, the developer who was reluctant (let's call him "A') in accepting the architect's schema design has to face the data architect again.

'A' rebelled and came up with a simpler easier domain specific schema design instead of the architect's generic schema design. He emailed the design to the team on the mailing list. A team member (let's call him 'B') spotted the inconsistency and asked the developer to reuse the architect's schema. This attracted manager's attention and the manager too asked 'A' to get the data architect's approval.

A meeting was setup to discuss the design with the architect. As expected, the architect and the manager opposed the idea of deviating from the standard, already agreed upon design. 'A' is very unhappy as he will have to now code a complicated solution.

This incident raises many questions. Should the scrum master intervene? He did not and he should not. Design is the team's responsibility and the scrum master should completely leave it up to the team.

What is the manager's role? Should he be involved? This is the grey area. In an ideal scenario, the manager should let the team figure out the design. But in large corporations, they often have policies and procedures which cannot be thrown away because one team is implementing scrum.

What should the team do? Other team members should talk to the unhappy developer and try to comfort him so that his productivity is not affected.

Day 2, Sprint 2 - User Story Granularity

We are in sprint 2 of a project. People are debating about the granularity of user stories. Last sprint user stories were very easy - e.g. come up with database design. Such issues did not require any coordination between the team members.

In this sprint we have chosen stories with multiple components. People are getting anxious as they will have to now depend on other people to finish their work. There is coordination involved. For example schema design has to finish before the web application functionality or service layer can be built. If schema design task of the same story is chosen by developer A and web application functionality is chosen by developer B, developer B has to co-ordinate with developer A. Developer B cannot start before developer A finishes his work. To avoid this, developer B comes up with cool idea of splitting the story into two stories - database design and web application functionality where database design can be scheduled in this sprint and web application functionality can be built in next sprint. Developer B calls it as a Common Sense Approach.
This common sense approach does not make sense because of following reasons:

1. Scrum emphasizes on self managing teams. It is very important that the team learns how to manage itself and how to collaborate with each other.
2. The use cases or user stories are supposed to be written more in use case manner. They must be business oriented. The stories must not be engineering driven, but strictly business driven. The stories in sprint 1 were not written correctly. They should have been written more in business manner.
3. First schema design, then serially building functionality of web application sounds more waterfally. Using agile processes schema design and building web application functionality should be doable in the same iteration.
   

Scrum Master called a meeting to explain why we cannot have stories like "Come up with schema design". He drew the following diagram :

He explained to the team that Scrum encourages to build a narrow piece of end to end functionality instead of artifacts such as schema design so that it can generate maximum business value. This also facilitates the feedback loop thereby giving product managers a chance to refine ideas on the product. People seemed to be convinced.

Day 1, Sprint 2 - Who does QA?

Sprint planning meeting went well but in spite of spending 2 hours and 40 minutes we could not allocate all the time at hand. We had about 700 hours at hand, including QA. We were able to allocate only 430 hours in 2 hours and 40 minutes. There were two QA persons. They ran out of all their time. How do we handle this? We had following options:

1. Scrum Master goes to QA manager and demands more QA resources for the project
2. Developers take up some QA work so that amount of QA work required can be divided in QA persons and developers.
   

Scrum Master mentioning second approach was met with wide eyes from the team! No decision was taken in the planning meeting. We had to stop the meeting because the conference room was booked then after.

The Scrum Master spoke with QA manager but QA manager expressed his inability to get more resources. The idea of doing QA was discussed a bit in the morning meeting, but since most of the team was not interested in testing, the Scrum Master abandoned the idea of asking the team to qa their peers work.

It was decided later in the day that the team will work on infrastructure or any other issues they want to work on in their remaining time. That work will be considered "extra work" and will not be demoed in the sprint demo.

In true spirit of scrum, I think we, the developers should have taken up job of testing their peer's work. But I can understand that this is not easy to sell it to developers.