The risks of large organizations in developing complex systems

• Authors: Robert Schaefer
• Venue: ACM SIGSOFT Software Engineering Notes
• Year: 2005

The risks to the large organization of being able to complete the development of large software intensive systems on time and in budget include not just the defining and maintaining of proper requirements and work processes, but also in the defining and maintaining of the proper organizational work structures. Bureaucratic structures prevalent in large organizations can be inefficient and irrational especially in regards to unplanned surprises, exigencies, contingencies, and the emergent properties that are the normal part of the development and integration of first-of and one-of-a-kind large systems. All large software intensive systems, at their true core, are experiments, and experimentation and bureaucracy have proved to be, over time, destructively antithetical in basic values. Conway's law, first stated in 1968, has by the lights of 2005 provided only a superficial explanation of the antipathy of bureaucracy towards the unknown and risk. The belief that "[o]rganizations which design systems...are constrained to produce designs which are copies of the communication structures of these organizations", can be easily refuted by noting that hierarchical organizations have been quite successful in designing and implementing networked systems subsequent to the invention of Ethernet and TCP/IP. The example provided by Conway was not sufficient by itself to draw the conclusion was made, a common fallacy of cause and effect, but he was certainly on an interesting path of investigation. Large system development failures cannot be prevented soley through improved planning, requirements, processes and software. Improvement must also come from the recognition of the limits of organization, planning and process; the recognition of the basic antipathy between bureaucracy and risk; the recognition of the need for organizational structures that scale yet remain responsive; the recognition of the need for better attitudes towards faults and failures, and the recognition that in order to reduce the risks that are inherent when the technical becomes political, that one must reduce the risks of speaking truth to power. The short article that follows is intended to raise more questions than provide answers, and the references are intended to provide a starting point for further research.