OpenView Development Forum Blog Post from Pivotal Labs

Ian McFarland and I attended a 2-day Development Forum hosted by OpenView Venture Partners last week in Boston. It’s the second year that Pivotal Labs has participated in the event. Open View has a portfolio of companies from all over (Europe, Australia, the US), each of which has been working on implementing Scrum over the past year. The engineering staff from 10 portfolio companies attended the event. Jeff Sutherland (amongst other things the co-creator of Scrum) is a Senior Adviser to OpenView; he provides advice and guidance to the portfolio companies as they progress through their Scrum adoption, and he gave a talk at the Forum. Pivotal Labs was invited to speak and lead a discussion of on 2 core topics: Developer Testing and the Principles of Build.

Status & Issues

First, we heard from each company on how they were doing with their adoption of Scrum and on the following two questions on the subject of technical developer practices:

  1. The goal of every sprint is to have fully tested quality software product ready to go live with customers. Where are you now with relation to this and what stands in your way of getting there?
  2. The goal of every development team should be to have one piece of software code globally, and all work and testing is done on this one piece of code, with multiple builds with built-in testing done in a given day. Where is your team in relation to this goal and what stands between you and getting there?

The 10 portfolio companies come from a disparate set of industries and technical domains, so Ian and I were very interested to hear each of their histories with Scrum and the current issues they were facing with respect to these 2 questions on developer practices. Some were at a fairly advanced stage – they had good test coverage and a stable CI setup – and some were just getting started. The most frequently stated barriers to achieving the goals of fully tested software that’s ready to release were:

  • A perception that the team wasn’t big enough to allow for writing tests
  • An assumption that ongoing developer testing could potentially slow the team down
  • An assumption that adding tests to a code base would mean first stopping and covering all the untested code so far with tests, and that therefore it meant stopping for a long period to retrofit tests
  • A question of whether clients even wanted frequent releases (e.g. once per week) of the software
  • A perception that the team wasn’t big enough to allow for pairing
  • A lingering assumption that QA and not Developers should be responsible for generating all test coverage

Pivotal Labs’ talks

Developer testing

Pivotal’s first talk was on developer testing. There were two main points we wanted to make:

  • The best way we know to get a big jump in quality and to be able to frequently release new versions of software is to rethink who is responsible for testing.

Rather than the traditional model of a QA team being almost solely responsible, consider a shift towards the whole team being responsible, and in particular a much greater emphasis on developers owning quality. For many developers it’s a radical shift.

  • The most effective way we’ve seen for developers to own quality is through the disciplined and sustained practice of TDD.

During the talk I gave a demo of a simple example of strict TDD, which gave rise to some useful conversations; as expected, the reactions varied from “yes, that’s what we do” to “that makes no sense!”. Having coded solely with strict TDD for almost 9 years now, and being around Pivots who exclusively test-drive also, it’s always interesting to hear reactions of people coming to TDD for the first time. The idea that tests are the center of the development effort, and that code is to some extent expendable, is a radical shift in thinking. We also touched on the benefits TDD brings in addition to reducing regressions. I find that a useful question to ask is what TDD stands for: “Test Driven Development” or “Test Driven Design”. The notion that TDD helps in designing your object model brought up some interesting discussions (for example, mock objects came up).

We also tried to address the barriers to adoption that had been brought up:

  • When introducing tests to a legacy code base, rather than trying to immediately cover everything with tests right away, we suggested:
  • “Stop the rot” – from now on, attempt to test-drive everything, including bug fixes
  • Spend time each iteration/sprint adding some coarse-grained, high-level tests around the legacy code
  • Once there’s at least a basic safety net in place, spend time refactoring towards unit-testability
  • Gradually introduce unit tests over time, with the goal being high test coverage at the unit level
  • Certainly with a small team there may not be bandwidth for an explicit traditional full-time QA role. We pointed out that with developer testing there’s no tester-developer separation: every developer is a tester, so testing isn’t gated by team size at all.

We made two points about speed when doing TDD:

  • It can certainly take some time to get proficient and achieve the same short-term speed you had before. However, in our experience, if a team has a member who’s done it before, or whose role it is to advocate for constant TDD, the team members start to get the hang of it much faster than you might expect – 3 weeks or less in many cases.
  • Independent of whether coding a certain feature is faster or slower, the medium- to long-term benefits (catching regressions instantly) are invaluable. There was an interesting discussion of why we’re so interested in pushing for frequent releasability even on projects whose clients are known not to want frequent releases. The consensus was that even if there’s no actual need or desire to release frequently, the act of trying to get to that point brings lots of benefits. There are fewer surprises when a real release comes due; the “last-mile” problem is often reduced; people integrate their changes with the main trunk of development more frequently, so there’s less merge hell, etc.


Our second talk was on build. As an ideal to shoot for, we promoted what Pivotal Labs does:

  • Checkins prompt a build (clearly, this helps to identify which changes broke what)
  • A broken build is an anomaly – teams should immediately stop and fix a broken build
  • Keeping the build fast is critical, so that it stays relevant
  • The build must be easily visible to the whole team. At Pivotal we have 2 large TVs on the wall that clearly show the build status of all our projects

There was a good deal of discussion over how to get started with a build. Some ideas that were brought up were:

  • You don’t need tests to have a build. Compiling and packaging your code with each checkin, and making sure everything works is still very useful information
  • If your build is flaky (i.e. some tests fail seemingly at random), immediately detach the flaky tests to keep the build stable. Work on the flaky tests in isolation. The team needs to trust the build’s status.
  • If your build is slow, break out a slower suite of tests and run it in a separate process that runs less frequently. Work towards high coverage with fast unit tests so that the status of your fast build is meaningful.

Hopefully our talks were useful. Certainly they sparked plenty of discussion! Thanks to Jeff Sutherland, Igor Altman and Steve Rabin of OpenView for inviting Pivotal Labs to speak.

View video