Hi Graham,

Just a thought: you may want to create a method for users to submit their replies anonymously (or at least not publicly), respondents may not to want to advertise how they do this such that other organisations can use this as an argument to disparage their quality procedures.

Also, it might encourage replies if you say what your research is for - are the aggregated responses going to be published for everyone's benefit? 

For background, my perspective is as an ISA, so these are exactly the questions I ask of projects I'm assessing / auditing, and it can sometimes be quite challenging to get honest answers! There's the "right" answer, and then there's what the evidence shows actually happens... I notice that (except for question 1) you haven't put "why" in your questions, I find that's really useful to find out - two organisations can have very different practices (e.g. for audit frequency), and the "why" question can reveal that both are correct for different reasons, they've built different mitigations into their processes - or indeed face different situations.

Sorry that none of that has answered your question! Except to add my two penn'orth that from my experience there is no "right" answer to any of these, it all depends on software complexity, maturity of the software, maturity of the team (i.e. how long that team have worked on that software), maturity of the organisation (i.e. how good / thorough / appropriate the software development procedures are), safety / reliability criticality., etc etc etc. I vividly remember the days when I was running two software "teams",  one of which was audited to death and in forensic detail, the other of which was, literally, one person coding in their bedroom on the other side of the world! But both were correct and appropriate for their respective application. 

Hope you get some good responses, they are interesting and challenging questions.

Thanks,

Andy

Andy Millar said:

Just a thought: you may want to create a method for users to submit their replies anonymously (or at least not publicly), respondents may not to want to advertise how they do this such that other organisations can use this as an argument to disparage their quality procedures.

Good advice Andy

 Graeme Furley if you enable private messages to be sent by anyone with an account for EngX then those interested can PM you instead of replying publicly in the forum. 

To PM another user, hover over their name and select Send Private Message in the pop up window, or click on their name to visit their full profile and then 'connect' from there. 

Hi Andy,

Thank you for your thoughtful response. Your insights, especially from your unique perspective as an ISA, are incredibly valuable, and I appreciate the time you took to share them.

I agree with your suggestion about allowing anonymous responses. I will amend my post so that people can email me privately.

As for the purpose of my research, I apologise for not making it clear initially. The goal is to gather the aggregated responses, anonymise them, and share the findings in a research paper. This paper aims to highlight current shortcomings in our field and suggest methods for improvements.

Your point about including “why” in the questions is well-taken. Understanding the reasons behind certain practices can indeed reveal that different approaches may be correct in their respective contexts. I’ll make sure to incorporate this aspect in my future inquiries.

Thank you for your valuable input. It’s given me a lot to think about and will undoubtedly enhance my research.

Best Regards,

Graeme

Hi Lisa,

Thank you for your response - this is a great idea, I will incorporate this into the post.

Graeme

Hi Graham, be mindful of posting  your email address in this public forum though. That's why we encourage the use of the PM feature in EngX instead

I have set up a google form which will not require your name, or email to populate your answers, and you will remain entirely anonymous: https://forms.gle/B8MkNGMqyEt2VhgK9

Thank you in advance for any assistance you can provide me

There's an old adage in the software world that quality can't be inspected in and it can't be tested in - it must be designed-in and it must be built-in. The ideal is that you use a developer who has a thorough understanding of the code and a thorough understanding of what it's meant to achieve. Everything else is just useful add-ons but won't of itself make up for rubbish generated at the earlier stages.

In my part of the world no change goes out of the door unless it's been reviewed by someone else. There are several layers of testing, but error still get out now and again. 3rd party stuff (anything from operating systems to databases to 3rd party data sources, in the usual situation where the supplies claims backward compatibility) is tested insofar as as our system passes regression tests when built on top of the new versions. Major version changes of 3rd party stuff results in a much more comprehensive analysis, upgrade and re-test. Our customers will routinely do their own acceptance testing before putting new versions into production,

   - Andy.

Hi Andy,

Thank you for your response. You’ve touched on some key points that resonate with my research. The practices you’ve outlined align with what I consider to be effective in software generation and management.

I find this quote interesting:

AJJewsbury said:

Our customers will routinely do their own acceptance testing before putting new versions into production,

In my experience with User Acceptance Testing (UAT) in engineering design tools and spreadsheets, I’ve noticed a diverse range of approaches. With your experience have you developed any 'best practice' protocols or procedures with your customers? How do you support them in ensuring these tests are conducted thoroughly, accurately, and within a reasonable time frame?

I appreciate your time and insights. They are invaluable to my research.

Graeme

I'm not sure about "best practice' protocols" - the state of the craft seems to be far from perfect still. We're getting just past the edge of my day-to-day work now, but I understand the we usually provide a list/description of all the intended changes (at least one that should affect that customer) so allow them to "focus" their testing (and manage any parallel changes - e.g. to user training or internal procedures). I don't think we'd try to influence the detail of their testing too much - at best they'd then be taking the same approach as we had during our internal testing - which isn't likely to find any problems we haven't found and fixed already. Variety of tests and letting them focus on the particular ways they actually use the system (which probably isn't entirely as we'd imagined) probably gives the most bang for the buck.

   - Andy.

Hi Andy,

Your feedback is really interesting, and opening lots of areas which I hadn't considered. Thank you very much for taking the time to respond to me, I really appreciate your thoughts.

Graeme