A Few Simple Principles That Help You Handle Complexity

We all know that planning fails when the complexity of the problem exceeds the capacity of the planners to reason about it. So the natural tendency is to fight the complexity so we don’t have to deal with it. This tendency is sometimes induced by fear: fear of not being able to oversee or handle the complexity. This fear has leds us to develop techniques that are commonly used to fight complexity. Think of techniques such as Isolation, Separation, Centralisation and Integration. And we are accustomed to use these techniques both in organizational design patterns (social complexity) as well as in technological design patterns (technological complexity). Isolation is a technique often seen in projects. We isolate the problem as much as possible so we don’t have to deal with complexity of the “whole”. This is one the reasons many projects fail. We use separation techniques when we organize something into domains or hierarchies. This is in fact a variant of Isolation. By separating into domains, departments, groups, teams, disciplines etc. we hope to have distributed the total complexity. We all know that in practice this leads to suboptimization. And then there is centralisation. By centralizing things they have to become standardized more or less. This is the payoff we need to get rid of the distributed complexity that was there before the centralisation. But we didn’t really reduce the total complexity with this technique. We just distributed it in another way. The total complexity is still there and pops out now in other areas. And finally, there is integration. We leave things as they are and fight the (coordination) complexity with integration techniques. This leads however to another type of distribution of complexity: we moved it into the integration domain. But it is still there.  Now let’s take a look at examples where highly complex “systems” are handled without any major problems. Inspired by nature we see that a flock of birds is self organizing around a few very simple principles. 1: All birds must know the same principles. 2: a bird must control the distance to it’s direct neighbors (not too close, not to far). 3: a bird must match it’s own speed with that of it’s direct neighbors.  So by having a few very simple principles, you can make an endlessly huge (or small) flock of birds act all in the same way. You can let them follow any goal or direction. (only one leader who gives the direction would be sufficient).  These principles of nature have more or less been copied into the design of the Internet. It operates autonomously. Compared to the flock of birds principes you could say: 1: All routers must know the same routing algorithm. 2: All routers must be able to calculate the distance to their direct neighboring routers, more isn’t needed. 3: All routers must know how many hops to make to get a message to the endpoint. You can find more on the Internet design in network topologies designed by Paul Baran†

Now what would happen if we organized work that needs synergy according to similar principles? Let’s give it a try: Principle 1: All workers must know the shared principles (we must therefore have a shared language, that’s one of the most crucial things to centralize!). 2: All workers must know the direction their direct neighbors wants to go to (nothing more, nothing less, so if you design this clever, you also solve information overload or information underload). 3: All workers must know the innovation or change speed of their direct neighbors. That’s about it. No more control needed (at least, theoretically). Sounds simple, doesn’t it? Maybe it is just that simple. Or to quote K.A. Richardson: “Each element in the system is ignorant of the behavior of the system as a whole (this implies we have to trust each other that the total job gets done partly by others).  […] If each element ‘knew’ what was happening to the system as a whole, all of the complexity would have to be present in that element” (this implies workers do not need to know the total complexity but only that of their direct neighboorhood).  Now how does this match to architecture deisgn? I promote that architects design distributed architectures where possible because that makes the total architecture a lot simpler. And only centralize parts for which no good alternative exists. This approach would prevent that we have to fall back on other techniques like isolation, separation, centralisation. Good luck if you want to experiment with it! Please let me know if it worked for you!