Software Engineering is a Social Problem

I subscribe to the idea that software engineering is often times a social problem and not a technical one. Almost a decade ago I gave a presentation on the open source development model to a corporate audience and this idea was at the core of the presentation. Steve Weber's The Success of Open Source does a great job explaining the details of why open source is a good model to emulate. It's the reason companies continue to put more focus on open source development models as a way to collaboratively develop software, despite noise about incorrect notions that it's chaos.

I find myself reflecting on this idea of the social problem occasionally on various projects. It makes me think about problems differently and gives me a different perspective on how software is actually developed and how to do it better. It's usually an indication that focus needs to be, well, refocused.

When I say social, I mean the interaction of the individual and the group. The human element: communication, time, cost, resources, ideas, and even egos and political agendas. This even extends to the expectations of the end user.

The human processes and the procedures are a social problem.

Technical problems would be, for example, trying to make something work that is not known to work. The tool. This means using or doing something that has never been used or done. To some extent, any new technology or new way of implementing a project has this. This is a risk that comes with any progression of technology, software or not. It may under-perform, not work as expected, or not work at all. This is often times confused with trying to work around an unforeseen problem. That is a social problem. See how I went full circle there?

There is no human procedure or process associated with a technical problem.

If we look at the IEEE Spectrum article on why software fails, both large and small, the reasons are plentiful:

• Unrealistic or unarticulated project goals
• Inaccurate estimates of needed resources
• Badly defined system requirements
• Poor reporting of the project's status
• Unmanaged risks
• Poor communication among customers, developers, and users
• Use of immature technology
• Inability to handle the project's complexity
• Sloppy development practices
• Poor project management
• Stakeholder politics
• Commercial pressures

Of the 12 reasons listed, all of them are almost entirely social reasons. For example, I don't see how one can read this story about a $100 million project that failed and not see it was entirely a social issue.

Pair programming, extreme programming, waterfall, agile, and every other software engineering methodology out there is an attempt to solve a problem and that problem is mostly how everyone works together to accomplish a goal. Sure, these processes are an attempt to make better,repeatable solutions based on technology, but it's an attempt to make better,repeatable solutions based on technology by humans. It's an attempt at a procedure and a process.

What programming language to use or what libraries to use or what hardware design to use is really just picking the tool. This can be done poorly and some of the tools used are more effective and efficient at successfully accomplishing a project than others. Experience and knowledge should make these things largely a known factor, but in some cases this is glossed over.

Ultimately, I believe software engineering itself is the actual problem that is trying to be solved on many software engineering projects. When more focus is not put on this fact, projects fail.