Mustafa Ozdeser said:

Agile methodologies, such as Scrum or Kanban, enable quick adjustments and continuous collaboration, enhancing adaptability and responsiveness. Conversely, traditional methodologies like Waterfall may hinder the team's ability to respond promptly to evolving project needs, potentially causing delays or inefficiencies.

Yes. And conversely waterfall methodologies can ensure that there is a clear structured approach which precisely meets the input requirements and doesn't go off-piste which can happen with agile / scrum. There is loads and loads to be said about this. There are no magic bullets, all solutions have advantages and disadvantages.

My particular interest is in innovation in safety critical systems, which really highlights the challenges - waterfall techniques don't promote innovation, agile / scrum don't promote assurance of safety / reliability. A few years ago I had the opportunity to do a research Master's on this issue which unfortunately concluded that actually it needed several PhD's to be done on this as the answer was both far from clear and remarkably little work had then (circa 2010) been done on it. Since then there has been some more research done, but I've yet to see clear processes / recommendations coming out of it (if anyone has I'd be delighted to know). I have my own opinions and experience, but that would take for too long to include in a forum post!

So the title of this thread would make a good basis for a conference, or several conferences, and indeed probably has...but very briefly: waterfall can cope with changing project requirements if the project team know what they are doing, oddly agile / scum can sometimes be less good in practice on this aspect (what they are very good at is responding to new possibilities in the system, which is slightly different), but again it very much depends on the expertise of the team. It can seem appealing to apply agile / scrum with the argument that it will positively impact the adaptability and responsiveness of a development team in the face of changing project requirements, but be careful that this is genuinely the reason and that it's not trying to justify the fact that the team just want to apply agile / scrum (they are much more fun than waterfall!). No techniques will solve problems unless the technique is managed in a way to make sure that it does solve those problems.

It is a good question!

Thanks,

Andy

My day job is software development heavy and I've probably been through a few different methodologies over the last few decades - and quite frankly I've not been overly impressed with any of them. They're all sold, like some new religion, as hailing some bright new world that's completely different from what went before, whereas most seem to boil down to the same basic process - any significant differences really only in scale rather than principle. Get some good people who not only know what they're doing, but also have a through understanding of the existing system they're working on and the end user's requirements, let them do whatever works best for them, and I don't think you'll go far wrong.

   - Andy.

I have heard a lot of push back against Scrum recently, saying that it makes things worse for the engineers.

1. It takes the concept of "agile" and dumps a load of process on top of it.  This keeps managers happy, and the engineers get fed up with the endless meetings.
2. The "customer" is never actually the customer.  They don't want to tie up one of their staff in endless meetings with their suppliers.  So the "customer" is actually a manager.  And the manager already has a plan saying what tasks will be done when, because that's what managers do.
3. The whole thing turns into an endless treadmill for the engineers.  At the beginning of the sprint, you're given a task and a limited time to complete it.  Fail to complete it on time, and management will notice.  Do well, and they will give you a bigger task next time.  It ends up a way to keep loading more work on the engineers, while telling them that they are being "agile".

Having started as a programmer in 1970 for an aircraft manufacturer who used what they had available, which was the waterfall methodology with documentation and change management using the same processes as building aircraft. Since then I have used most of the development and management methodologies in a wide variety of industries during a career which has included computer manufacturers, consultancies and a wide range of UK and international companies. Anyone remember "goto less" programming, decision table programming, the forthcoming ice age and power so cheap we will not meter it? 

My conclusion is similar to Andy's above. Each new methodology is treated with the reverence of a religion with  zealots ensuring that all shall obey the commandments. In many cases the actual project plays second fiddle to the methodology.  I have worked on a project with what was one of the big six where the success of the project was based on the height/weight of the paperwork produced using their methodology. We were using a virtually self documenting 4GL and the relevant paperwork could have been contained in a small A4 binder, they actually produced over 3 feet of it. On one project I had to review a functional specification the customer had to sign off ,produced by a major consultancy. iI was 6,000 pages for an accounting system. Apart from me being unable to recognise what sort of system the specification was describing it did not mention project numbers around which all the companies finances were structured. I asked a friend in the project team and they said because it was complicated they had not yet worked out how to do it.

The successful development projects I have worked on have normally worked due to the commitment, knowledge and drive of a small team of the right people managed by  team players who can manage the politics of the organisation. The methodologies were irrelevant. Where there is a company wide project management team they must also have the right approach to ensuing compliance without stifling innovation. I have worked for a company where blame management was so rife even if you did the right thing you got the bame, projects did not go well.

Finally, for my attempt at proselytising: I would say that a successful project needs to start with the data and entity relationship modelling is a good way of doing this. If the underlying data structure is right then changes to process etc are easier to make irrespective of the methodologies used.

There is a significant issue here - you have assumed waterfall and agile are mutually exclusive. This isn't correct. Agile relies on a set of independent tasks, but something has to have determined the rough scope such that the you can do the agile assessment. This essentially is a little bit of waterfall - just enough so you can get started.

Even in a agile sprint, you are going to be refining those requirements so you can implement them, implementing them and then testing what you have done. This is a mini-waterfall in itself.

So there is actually nothing that stops you from taking concepts from both, but it has to be done for the right reasons, being careful to avoid some kind of wagile approach that combines the worst of both.

The other thing hinted by the original question is that waterfall is a one-hit approach. It isn't. I can't think of any major program I've ever worked on that was a one-hit. All of them had phases accounting for iterative and incremental growth of the functionality. Iterative and incremental are also one of the key aspects of agile.

However, change can be an issue irrespective of what approach you take. The real challenge here is knowing the impact of that change and that is all about traceability. A minor change may only result in the requirement, code and test being changed. A more significant change, such as changing the microcontroller in a product during development, can have significant impact in all areas that you might have start many things from scratch (been there, got the t-shirt).

With those "agile" projects I have worked on, the common issue was that pure agile assumes everything is independent. For complex projects this is rarely the case and as any Systems Engineer will tell you, there is often unexpected emergent properties of a system. Hardware is ultimately finite (especially in embedded systems).

So, my answer to the original question would be, the choice of methodology may seem to impact responsiveness to change, but ultimately how fast you can change is the wrong thing to measure, what you should be measuring is how well you are meeting the time, cost and quality targets of the project. None of the methodologies by themselves give you that answer.

Mark

Mark Tickner said:

So there is actually nothing that stops you from taking concepts from both, but it has to be done for the right reasons, being careful to avoid some kind of wagile approach that combines the worst of both.

The latest railway standard for safety related software (EN50716 which replaces EN50128) has a very sensible and pragmatic informative section on the use of iterative lifecycle models in a world which has - at least according to the standards - traditionally been very waterfall. Which is good, and a relief, for years (30+ in my case) as an industry we've been trying to pretend that we develop safety critical software (and, in fact, systems and hardware) in a waterfall form whereas in practice real R&D rarely is. And similarly we have occasions where we want to bring in software developed for other applications which simply won't have been developed waterfall. (I'm leaving aside COTS and even worse SOUP which is a whole other problem - I'm just thinking about software where we do know how it was developed.)

So it seems that the standards are, belatedly, catching up with your comment, which I totally agree with. Don't get too excited, it doesn't go into huge detail, but it's a good start.

Mark Tickner said:

but ultimately how fast you can change is the wrong thing to measure, what you should be measuring is how well you are meeting the time, cost and quality targets of the project

Even in the high reliability / safety critical world I'd put it slightly differently. I'd say that these are all KPIs, i.e. responsiveness to change should be included alongside meeting time, cost and quality targets. I am coming at this from a rail industry point of view, where traditionally our responsiveness to change, including the adoption of new technologies, has been so slow and risk averse that it could be argued that opportunities for improving quality, reliability and safety have actually been missed. But equally I don't like the approach of just keeping launching test rockets until they stop blowing up...as you suggest it is possible to combine innovative development techniques with sound engineering practice.

Cheers,

Andy

In reality, nobody would use waterfall in situations where requirements are subject to change.  And most people wouldn't use a strict waterfall at all for a big project.

There will most likely be some wiggle room in the development process to handle changes.  A couple of decades ago, Rational Unified Process was popular.  It defined a series of iterations, during which you work towards the finished product.  The iterative model allows time to fix problems found in the early iterations.

I am not too comfortable in mixing ‘software engineering methodology’  and the resultant debates about waterfall vs Agile and Scrum and the like as this mixes up structural dimensions. To me, the critical decision in software engineering/development is the language chosen, and this is dependent on what sort of software is being developed. IMHO,  software is grown, over time, from its initial capabilities onward. In aeronautics, though of course, the product, an airplane, does not fly a little bit. Software can be improved, expanded/extended over time. Also there is a great difference from web based, HTML ‘software’ and industrial strength compiled software that are more fundamental than project management methodologies, Agile,Waterfall and the like.