Psychology of Software Development

In 1987, Tom DeMarco and Timothy Lister wrote Peopleware, and it goes like this:

The major problems of our work are not so much technological as sociological in nature.

25 years later, it looks still true. I have mostly worked around open source, where I do recall big public clashes, like gcc versus egcs, or glibc versus eglibs, caused by social reasons. Ongoing systemd saga is mostly social. Everywhere, design discussions become personal ones, with no way to decently get out. Regardless of technology, good maintainers often have particular character traits. Now that's just me, and the world of open-source and cute nonsense hacks. Do real psychology researchers have anything to say about real professionals?

Already in 1975, Flow and Intrinsic Motivation began to question classic ways to run a business. Since then, psychologists found many pieces of the puzzle, showing that sociology and psychology are indeed important. Managers often don't care. Waterfall was sometimes replaced with agile, with its "Individuals and interactions over process and tools", but that soon became a new cargo cult. In the original curgo cult (Surely You're Joking, Mr. Feynman!), natives sat with fake wooden headphones, waiting for airplanes with goods to arrive. These days, we sit with actual headphones, hold all the right meetings and wait for team to "self-organize" and for the breakthrough to happen.

Let me try to give an index of current psychology that apply to software engineering, so that if you want to a better software developer, or engineering manager, you can have an easy starting point. The material here follows the SECR 2014 talk I recently gave, but is edited to be more hands-off and is in English.

The puzzle pieces

We like to think that evolution meant to create humans all the time, but it's not a smooth directed process. Rather, it's a random competition between every two different genomes - including between viruses and animals, animals and humans, humans and other humans, and even humans of different genders - with no ultimate winner (The Red Queen). Multiple behavior strategies were formed in humans as the result, with emotions to activate them, and all of us have a custom mix of them to use. Each strategy is good enough, statistically, to stay in human genes, but no single one is guaranteed to work for any particular human. Worse, these strategies were created to work in prehistoric environments, which no longer exist (The Moral Animal). No wonder that we're driven by emotions, and nudges, and often make wrong decisions.

We know that all kinds of emotions are important to work, such as motivation (Switch), happiness (The Happiness Advantage) as well as overall ability to deal with emotions in oneself and others (Emotional Intelligence). Doing the right things is easier with right habits (The Power of Habit) and with helpful nudges (Nudge), but at times, willpower need to be involved (The Willpower Instinct).

Behind the surface, our brains are nothing but neural networks, and those are not perfect. They can quickly learn, and then respond very quickly, basically automatically. The clusters of strong neuron connections created by learning, called attractors, get stronger each time they are used, and we get progressively efficient. But that's only when learning has a proper feedback loop - fast, positive and consistent. Otherwise, we form false attractors - that fire fast, and get stronger with each use, but produce wrong results. We're particularly prone to create false attractors when feedback is slow, and probabilistic.

For thinking, false attractors are responsible for thinking biases (Thinking Fast and Slow), such as coming up unrealistic estimates, siding with other's unrealistic estimates, picking up first design that comes to mind without checking, and trying hard to avoid past mistakes that are unlikely to happen again. For emotions, seriously false attractors require attention of a psychotherapist, while in common case, we can get excited or upset just because we were in similar situation before, not because the response makes sense today (General Theory of Love). At the intersection of thinking and emotions, we often get excited about rewards, and regret later. One classic example is quarter-end rush to sign deals - and then figure out how to get them done. This is one case where introverts can be extremely helpful at work, since they don't get excited about rewards all that much (Quiet). We do have a capacity for slow, more formal and symbolic thinking, but it's much harder, and the need to use it often can only be pointed by others.


The picture

There is no final picture, and many pieces don't fit. Many are broken, despite passing statistical tests, because most published research findings are false. Many are discolored, having so small effect it's not clear where to put them. The sampling bias is caused by using mostly students for experiments. The measurement bias is here - say, how would anybody measure programmer's performance? I would hope nobody counted lines of code written or tickets closed.  And many psychological theories are just not good scientific theories, if we use the definition that Steven Hawking gives in Brief History of Time:

good theory [...] must accurately describe a large class of observations on the basic of a model that contains only a few arbitrary elements, and it must make definite predictions about the results of future observations

Surely physics cannot predict the future of entire universe, but it is fairly good at predicting where a rocket would go if an engine is fired. Psychology, in contrast, has a large set of observations but struggles with useful predictions or interventions. Influential books are particularly at fault, explaining how flow, and happiness and emotions are important, but failing to mention that finding flow in gardening, and walking around smiling might not be great strategy for a person or society. Evolutionary psychology (The Moral Animal) holds some promise here, having small foundation, and trying to predict everything, but it's too early to judge its success.

The corners

Not having a complete picture is unfortunate, but looks like the corners are filled in - we know few important areas where neglect is detrimental.

• Emotions. Emotions matter, and while Real Professional can do unpleasant things for a while, without emotions, habits and nudges, he will likely give up. I'd conjecture that often, team opinion and motivation about possible new features is more important than the estimates of market value.
• Trust. Is everybody in the same boat, heading in the same direction, sharing the same prizes, and actually rowing when needed?
• Feedback. For automatic thinking to work well, we need feedback that is positive, timely, and consistent. That does not happen by itself in complex areas, say quality, or architecture design, and need to be explicit part of the process - be it review of key architecture decisions few months down the road, or having low defect count celebrated.
• Slow thinking. Automatic thinking will eventually make mistakes, and then start repeating them.  If you have orderly predictable Scrum process, you can easily become too complacent. There must be time to make complex technical decisions, and time to review direction from the basic principles.

Do these corners receive systematic attention on your team? Are they among process metrics? Probably not, and probably some of them need improvement. Lacking full picture, we don't know how much effect any changes will have. Maybe, things are already not too bad in ad-hoc way. Maybe, some are totally broken. Anyway, we're software engineers, we're good at setting up processes, and if we keep these four corners in mind, we can make our processes work together with our psychology, and not against it.