Like many collections of works, the quality in this volume varied. I only remember one of the articles being particularly bad. A worthwhile number stand out as good.

The first four articles cover how to read software engineering research. I wish that I had read it in grad school. If you find yourself reading academic papers in computer science, it's worth reading these four articles.

The rest of the articles cover research about different areas of software engineering. If you're like me, your opinion of each article will be partially influenced by its quality and partially influenced by your interest in the topic. That said, there was a general pattern that the essays that tried to very narrowly investigate whether or not some piece of common sense wisdom were supported by evidence were, simultaneously, the best research, in terms of not overreaching, and the worst reading. :-) ( )

Not nearly as much meat here as I was hoping for. Indicative of the problem, it is hard to measure software development and hard to interpret the data that exists. ( )

An important read for everyone in software development. Although the book is not executed perfectly, it raises the level of debate in the software industry from anecdotes and opinions to hard data and research.

The second half of the book is a great collection of research results across a variety of important software topics, such as:

* Learning programming: Why is it so hard? Do better tools or visual programming help?
* TDD: Does it reduce bugs? Does it lead to better design?
* Pair programming: Does it reduce bugs? Does it increase or decrease productivity?
* Code review: Does it reduce bugs? Should you do it in groups or individually?
* Women in computer science: Why are there so few? Is it due to genetic differences or cultural biases?
* Team organization: Is Conway's Law something to avoid or embrace?

This book is now my go-to source for a variety of software decisions. I just wish I had read it long ago.

The reason for 4 stars instead of 5 is that the way this information is presented is not particularly compelling. Most of the chapters in the first half of the book, and a couple from the second half, are written in a dry, academic style that's too focused on the nitty gritty details of software research methodologies. I suppose that's OK if the target audience is other researchers, but my impression is that the goal of this book is to bring evidence-based software engineering to the typical programmer, and to do that on a large scale, you need a much more approachable writing style. In other words, if the goal of this book is to motivate change, then the authors need to pick up a copy of "Made to Stick" and learn to simplify the message (e.g. gloss over the research details), make it more concrete (e.g. explain what it means in the real world), involve some emotion (e.g. these are controversial topics, feel free to make some jokes or have an opinion every now and then), and tell stories (e.g. give examples of how these results affected an actual project).

Overall, a very worthwhile read, but if you're not a researcher, be prepared to do a lot of skimming, especially in the first part.


Some good quotes from the book:

We hope the questions and answers in this book will change how you think about software development. We also hope these essays will persuade you to say, “Citation, please,” the next time someone claims that one way of laying out braces in C or Java is better than another.

Convincing evidence motivates change.

Evidence is not proof. In general, evidence is whatever empirical data is sufficient to cause us to conclude that one account is more probably true than not, or is probably more true than another.

Qualitative research has to precede quantitative research and will look at situations that are more complicated. When only few different factors are involved (such as in physics), one can proceed to quantitative investigation quickly; when many are involved (such as in human social interactions), the transition either takes a lot longer or will involve premature simplification. Many of the credibility problems of software engineering evidence stem from such premature simplification.

We found that programmers deviated from a reference group in that they are lower on Extraversion, lower on Emotional Stability, and higher on Openness to Experience. [...] Programmers are also more homogeneous than the population as a whole; that is, programmers vary less in personality than do people in general. This confirms the stereotype of programmers being neurotic, introverted, and intellectual—and, by the way, male (which I know for a fact some people consider tantamount to a personality trait!).

It makes a significant difference whether you ask someone how much time he needs to complete a given amount of work, or whether you ask how much work he can complete in a given amount of time.

Thus a possible corollary of Conway’s Law is: A software system whose structure closely matches its organization’s communication structure works “better” (defined broadly) than a subsystem whose structure differs from its organization’s communication structure.

Every page in this book has been checked over by an editor. Why? Because even if you’re the smartest, most capable, most experienced writer, you can’t proof-read your own work. You’re too close to the concepts, and you’ve rolled the words around your head for so long you can’t put yourself in the shoes of someone who is hearing them for the first time. Writing code is no different. In fact, if it’s impossible to write prose without independent scrutiny, surely it’s also impossible to write code in isolation; code has to be correct to the minutest detail, plus it includes prose for humans as well! (You do write comments, don’t you?) ( )

As a collaborator of this book my review would be quite biased, so I'll refrain from judging it publicly, and from rating it. But I will suggest that perhaps the best way to read it is to browse through its contents first, to get a sense for what is discussed in each chapter, and then to use the book as a reference to find out what researchers think about the topic you're facing at the moment.