Agile Project Management: The Battle-Tested Field Guide
Nov 5, 2020
Software projects differ from projects in other engineering disciplines. For instance, if you think about a construction project that's late and off schedule, you'll (rightfully) think that adding more construction workers to the pool of employees working on the project will speed things up. The same happens for software projects, no? Well, no. It doesn't. As described in the seminal book The Mythical Man-Month, "Adding manpower to a late software project makes it later" (p. 25). How can that be?
This post is about agile project management in software projects and will try to answer this question, among others. I'll explain how agile has evolved from previous project management approaches, what its main building blocks are, and how to manage agile software projects in the best way possible.
Prehistory: Before Agile
Actually, agile project management is a relatively new approach. Although the roots of the ideas behind agile project management can be traced back to the 1970s, the approach itself became popular and widespread (and, we might add, even dominant in the marketplace) less than 15 years ago. The conventional way of managing software projects before that was the waterfall model.
Managing Software Projects the Waterfall Way
According to the waterfall approach, the right way to manage a software project is to move it sequentially through its different stages, from receiving the product requirements from the client, through the actual development phase, and up to providing maintenance services for the delivered product. This approach stems from construction projects. They're still run this way up to this day. At the beginning of the software development industry, this model was copied without too much thought.
Waterfall Model in Detail
Project/Product Requirements
In the waterfall model, every software project starts by understanding the full scope of what the clients want to build. We call this the requirements phase. In this phase, we're talking about business requirements only. We'll try to answer questions like these: What does the software have to do? Who will use it? What different layouts will be in the system (if it contains a GUI)? What are the main features?
Evaluation
Once the requirements are finalized, we need to estimate the cost of the project. This is done by estimating the number of developers, designers, product managers, and other involved individuals and taking our best guess regarding how many of them we'll need. And how much time it's going to take them to develop the project and hand it over to the client.
Software Architecture
After we've understood the needed manpower for the project (in software projects, rarely do we speak about other resources than manpower—we don't need cement here) and, in the case of an external client, they've signed off our quote, we can start doing the software design. In this phase, we do the high-level architectural design of the software we're going to develop, and we'll want to answer questions such as these: What databases are we going to use? Back-end/front-end frameworks we're going to use? What will be our data model? What are the main entities in the system? The deliverables of this stage are technical documents such as data-flow diagrams (DFD), entity relationship diagram (ERD), and UML charts.
Coding
In this stage, we do the actual coding. We implement the technical specification in the form of classes and actual software code that does things and moves bits and bytes.
Testing
After the coding stage, we need to check that the software created meets the specified business requirements set by the client (and sometimes the technical requirements as well—set internally).
Deployment and Maintenance
After the development, we need to deploy the software for the client. After the deployment, we need to verify that everything works properly. In addition, sometimes as part of the service agreement, we provide the client with monitoring, support, and maintenance services for malfunctions and disruptions that occur in the system.
Problems With the Waterfall Model
Software Development Isn't Like Construction Engineering
There are fundamental differences between software engineering and construction engineering.
No One Knows How Much Time a Large Project Is Going To Take
In construction engineering, a project may be behind schedule or ahead of schedule, but this is regarded as an anomaly. A situation that must not happen and, in most cases, doesn't happen. This isn't the case when we discuss software engineering. Most of the projects developed in the waterfall model are substantially behind or ahead of schedule. This is the normal way things go in the software world. This happens because it's really hard to translate business requirements into working software code. In construction engineering, the methods, processes, and materials are all well-known and established. Software engineering is a new and rapidly evolving discipline lacking those "know-hows."
Cost of New Requirements
One of the motivations behind this approach in the construction industry is the cost of change requests. The cost of change requests is prohibitive. Just imagine the client asking for a third floor after we've just placed the roof on top of the initially requested two. This is a very expensive change. Software requirements that are coming late in the process can be expensive too but not nearly as much as late requirements in other engineering disciplines.
Adding More Manpower Doesn't Shorten the Project
When the number of developers grows, so does the overhead of managing and streamlining communication. In addition, when you add a new developer, you need to have them up to speed with the project requirements, the architectural requirements, and the current state of the code and explain to them their specific task. This is a long ramp-up time that is very costly. In addition, it diverts existing senior developers from coding new features to mentoring new developers, thus diminishing their output. Adding many new developers to an existing project to speed things up will actually make the project be delivered even later.
Here Comes Agile
Agile software development takes a new angle on software projects. Instead of living with late projects and hoping for doing better estimations next time, the agile approach recognizes the specific and unique nature of software and takes a different path. Although the roots of the agile development movement can be traced back somewhat to the 1970s and surely to the early 1990s with the rise of such similar approaches as iterative development and rapid application development (RAD). The approach really started to gain traction after the agile manifesto was published in 2001 by people who were acutely aware of the limitations of the waterfall model and wanted something better for themselves and the rest of the industry.
Agile Principles
The agile approach encourages working in a dynamic, iterative way. Instead of defining the whole scope ahead of time and doing everything once, we focus on development sprints and quick feedback loops. In other words, we don't know how much a whole project will take, but we do know what we can achieve in two weeks with our current manpower.
As stated above, we work with sprints and quick iterations. We get a list of features that we need to finish in two weeks, we do the software design accordingly, and after those two weeks, we deliver a working, operational software.
Agile Advantages
There's a much greater chance of delivering software on time this way. And there's a much greater chance of delivering working software. The feedback loop from receiving client requirements to having a working software delivered to them shortens significantly. Instead of waiting months or even years to see "something," the client can give feedback in a matter of weeks. Thus, if any changes are necessary, they can be addressed early in the development process. Thus lowering the cost of each change and the time it takes to implement it. The main benefits and outcomes of this approach are outlined in the 12 agile principles of agile behind the agile manifesto:
Working software
On time
With quick feedback loops
Can accommodate change requests fairly easily
Results in the ultimate goal of high customer satisfaction
I'll devote the rest of the post to properly managing projects the agile way. Note: the agile approach is a broad term that encompasses several subdivisions or "process frameworks" such as scrum, extreme programming, etc. In the following sections, I'll describe the main components of developing software projects the agile way. I won't elaborate on each components' originating process framework (scrum, extreme programming, iterative design) as the broad principles are mainly the same and the exact origin isn't relevant in day-to-day work.
Agile Project Management
Sprints
One of the basic building blocks of the agile approach is the sprint. The sprint is the basic unit of work that's done by a development team. The product manager prepares the client's requirements in the form of features that the team will develop. Each feature is a collection of separate user stories. Those stories combined should implement the feature requested. Before the sprint, the product manager should sit down with the team leader/software architect to evaluate the stories and make sure the team will be able to develop them during the sprint. A sprint's length is usually two weeks, but sometimes companies adjust it for their needs to be one to three weeks. It's important to allocate some of the time of the sprint as non-productive/non-working time. Some team members might be sick, an urgent production bug can pop up, etc.
Presentation
Each sprint starts with a presentation to the team members. It's important that everyone be on board with the stories they're going to develop. In addition, any business/technical questions should be addressed. In this stage, stories are evaluated for length and complexity, and effort estimations are given. All the stakeholders are present in this meeting: software architect, team leader, project and product manager, developers, QA engineers, and DevOps engineers.
Planning
We follow the presentation stage with a planning stage. In the planning stage, we break down the stories into subtasks and understand how to best implement them from a technical standpoint. Usually, a senior developer/software architect will be present during this stage.
Day-To-Day Sprint
A development day starts with a "stand-up" meeting (originated in the scrum approach but used in agile as well). We call it this because it's around 10 minutes tops, and all the participants are standing (no comedians involved). The purpose of the meeting is to get a recap of everyone's work the day before and their plan for today. This is the proper place and time to raise questions and mention if something is blocking a developer from developing their user stories.
Non-Productive Work
As mentioned before, time should be allocated for unknowns and general unexpected things that might happen. For instance, production bugs should be addressed immediately as they affect the end users of the developed application. The team leader and project manager should be alert to such occasions and make sure that they'll still be able to deliver the sprint's user stories on time.
Deployment
After developing and testing the stories, we need to deploy to production servers. It's important that DevOps engineers/SREs are present during the presentation stage and available for consultation during the sprint. This way they'll understand the context of what the developers are working on. This understanding, in turn, will help them quickly fix any operational issues that might arise during development. In addition, they can do the necessary preparations for rapid deployment of the developed features.
Retrospective
Another key component of the agile/scrum approach is the retrospective. After each sprint, the team members meet to discuss the sprint that has just ended. The purpose of this meeting is to create an iterative, continuous improvement of the team's performance. The meeting revolves around those three main points:
What went well in the sprint?
What can we improve?
And what specific action items will we take to achieve this improvement?
As stated correctly here in this detailed explanation of what a retrospective is, "Although improvements may be implemented at any time, the Sprint Retrospective provides a formal opportunity to focus on inspection and adaptation." In the next retrospective, the team members will assess whether things that should have been improved really did improve and, if not, whether the action items needed for this improvement were done.
Scrum Master
A unique role for the agile/scrum process is the role of the scrum master. Usually, this role will be assumed by one of the team members. If the team has a release manager, this person will usually be the scrum master as well. Otherwise, there's no preference for who will take this responsibility among the team members. It can be a developer/QA engineer, the team leader, etc. The main responsibilities of the scrum master are to be in charge of the scrum process. Make sure that everyone arrives to the daily stand-up meetings on time, host the retrospective meeting, make sure that everything is clear during the planning sessions, etc. In addition, the scrum master is often the main point of contact between the team and the organization. The job is basically that of a "process streamliner."
Taking Agile To the Next Level
An industry that hasn't been disrupted yet by the agile revolution is that of enterprise companies developing software. However, this is changing today as well, as agile gains more and more ground in the enterprise world with companies like Plutora that provide a complete solution for adopting agile methodologies in enterprise organizations.
The Plutora platform essentially allows full visibility and insight into the value stream management (VSM) process, as described in this book on VSM. Aside from VSM, another way to pursue agile is considering digital transformation for your organization.
Conclusion
Though its roots can be traced back 50 years, the agile revolution started to gain traction only 15 years ago or so. It became dominant and a standard in most industries only in recent years. Today, with companies that provide a whole end-to-end platform for facilitating agile development, it enters the enterprise market as well. Thanks to its proven benefits of delivering better software on time, it's sure to optimize the way enterprise software is developed.
Download our free eBook
Mastering Software Delivery with Value Stream Management
Discover how to optimize your software delivery with our comprehensive eBook on Value Stream Management (VSM). Learn how top organizations streamline pipelines, enhance quality, and accelerate delivery.