The Hollow Hype around DevOps

Tonight was another gathering of the nlscrum crowd. The speaker today was someone I had seen present at devoxx before, Patrick Debois. I left his session at devoxx quite unsatisfied, so I figured I’d take the opportunity of this smaller group to see what his brainchild, DevOps, is really all about.

The presentation Patrick and his co-host gave left me unsatisfied yet again. Patrick is stating that there is a big problem between Developers and Operations. Dev always think that Ops is not flexible enough and Ops always say that Dev are cowboys. Admittedly, Patrick is a good presenter and brings a nice story. He tells a story from his development past about building an application that would occassionally crash and eventually burns to the ground. His point is that there is a big divide between how a developer thinks and how an operations person thinks. The differences come from the following I think:

Developers Operations
Focus on Change Stability
Are responsible for New Functionality Non Functional Requirements
Work in Iterations Flow
Use process agile ITIL

My problem is that in both his presentations Patrick only puts down a problem and the message is that we should change our mindset. We should embrace each other and learn to think alike. That’s really, really not tangible enough for me. I feel the problem since I am in a role as both Dev & Ops to an extent and want to get some good advice on how to combine the two better. So far, the 2 DevOps presentations did not deliver.

At the end of our meetup, we always have a lot of room for discussion. I put a topic on the board which was meant to lure Patrick and his co-host out and into the open. I titled it “The Hollow Phrase – DevOps”, hoping to get a good discussion going. It appears I was being a little too Dutch and straightforward so my fishing hook didn’t directly receive a bite. I managed to find Patrick anyway and opened up the discussion with him and a few others. It turns out, there are a few tangible things we can take away when we think of having Dev and Ops play nice.

Lessons for Developers

Focus on Non-Functionals

Developers should really have Non Functional Requirements in mind when building an application. This sounds like a real no-brainer, but in most projects and teams, important things like scalability and performance are only an afterthought.

There is however a practical way to make sure that this is considered core and also gets attention: make sure the product owner has Non-Functionals in his field of vision. Plenty of ways to do this right:

  • in smaller organizations, have the end-customer define what level of pain they are willing to take
  • in bigger organizations, make sure that an architect from the Operations department is a regular partner for the product owner

Share the responsibility

The problem gets worse when Developers are only responsible for delivering new functionality and Operations are only responsible for maintaining stability. Make sure both parties are responsible for changes and stability. Developers can also be held accountable for the number of incidents on their software after release. Operations can also be held accountable for the number of successful changes they manage to implement. In organizations with bonuses, it pays off to define the bonuses on both these levels.

Handling incidents in iterations

In organizations, such as Amazon (and Compare Group šŸ˜‰ ), the developers are also responsible for the operation of the software they build. The problem you run into with this combination, is that your Sprint commitments are endangered the moment you run into an incident on your live environment. The solution is simple: assign incidents a value, just like the stories on your Sprint. This value can be in story points, in hours, whichever works for you. At any given time your commitment stands to deliver a certain number of value points in a Sprint. The only discussion you need to have is: what do we take out of the Sprint now that we have run into this. In my opinion, this should be a natural part of the negotiation between Product Owner and Scrummaster anyway, so use that mechanism.

Lessons for Operations

Use virtualization and cloud technology

New technologies such as virtualization can make an Operations team’s life a lot simpler. No longer do you have to be reliant on this one server, but in case of failure you can simply move your virtual machine somewhere else and be guaranteed it will work again. Also, this type of technology facilitates scaling a lot better, since it’s extremely simple to clone environments.

Infrastructure as code

Work on repeatable and durable solutions, stay away from quick one-time fixes. A good way to do this is to make automation a key part of your Operations. Setting up a server is something you can have a script do for you. Write the installation process once and then run it twenty times. Automation not only saves time in the short run, it also prevents a lot of errors in the long run because there are a lot fewer places for errors between chair and keyboard.

Empower the business

Delegate recurring change requests to your customer. Simple change requests still take up a lot of time of Operations teams, the customer is quite capable of doing them himself if he has the necessary tooling. In many cases you will need to make a user interface available for your customer, but after that he is in control. He won’t need to get in touch with your helpdesk anymore to get his password changed, he can just receive a new one in the (e)mail. This frees up your time as Operations to work on longer lasting changes.

Conclusion

DevOps has some abstract points. As Developers it is important to think about what your software will do in a production environment. As an organization, it is important to make sure that the responsibility of the Non Functional Requirements is not guarded solely by your Operations team.

There are also some concrete points to take away, but that requires peeling away some layers of the onion. That’s really a shame, since Patrick and both his co-hosts had good looking presentations that had obviously received a lot of work. Still, it was good fun to discuss this with Patrick and his co-host and I have every bit of respect for what he has accomplished so far.

Advertisements
This entry was posted in agile, development, scrum. Bookmark the permalink.

7 Responses to The Hollow Hype around DevOps

  1. Erik Buitenhuis says:

    You mention agile as the process choice for DEV, ITIL for OPS. In my opinion Kanban (agile) would suit OPS very well. In teams that mix OPS and DEV (you named Amazon) Kanban could also be a suitable process.
    Erik Buitenhuis (CSM and CSP but no Kanban certification šŸ™‚

    • sunsear says:

      It was definitely mentioned during our discussions that KanBan would suit Operations very well. In general one can also say that KanBan could suit Development very well depending on the organizational needs. It would be easy to make a combination there and connect the 2 teams together that way. We discussed this with Patrick, but he advised against it. His experience suggests that having really large KanBan boards means that people will have a lot of things on the board that they do not (yet) care about and they lose interest. He advised having 2 boards, 1 for Dev, 1 for Ops and then putting the output of Dev to the input of Ops.

      Martin
      CU (Certifiably Uncertifiable)

  2. Pingback: Kristian blogt » De strijdbijl tussen engineering en production begraven?:

  3. Erik Buitenhuis says:

    Fully agree on that approach. Having lots of stuff on the board is bad because most of it will be irrelevant to you; they’ll just obstruct your view on the things you do care about. So if you have 2 teams, have 2 boards (and if 1 team does all, have 1 board). Connecting the boards makes sense. Reminds me of a presentation by Serge Beaumont at last year’s XP days where scrum boards for product owner team and scrum team are connected. When the PO team moves a story to “done” the scrum team can start on it (the story is “ready” to implement – according to a certain “definition of ready” similar to scrum’s definition of done).

  4. Mary says:

    I’m neither a DEV nor an OPS and understood that the basic NFR’s were translated in the Definition of Done.
    When OPS qualifies reoccurring incidents based in, lets say Quint2, they have a clue what makes incidents occur and realize they don’t have to handover the same problem more than once to DEV.
    My conclusion of the evening is that live & work could be much more enjoyable and fun when people (of any feather) respect each others craftsmanship and work in multidisciplinary teams.
    Kind regards
    Mary

  5. Cees de Groot says:

    We’re starting to move into the DevOps way as well. The basic tenet is healthy: the team should be responsible for the whole value chain, and as you want to put your stuff into production to reap actual value from code, the team needs to take responsibility there as well. I’m actually happy about the “why” being clear and the “how” being abstract – even though it is nothing really new, ideas around it seem to be in the process of being formalized only now, and getting too much concrete guidance too early would freeze experimentation.

    At our shop, we’re looking at having an infrastructure part of operations responsible for supplying teams with hardware, networking, etcetera. Everything up from (in our case) the JVM then becomes responsibility of the teams who have embedded operations people. It’s not working as well as it could be (nothing new there – change needs time ;-)), but we’re making headway and the most important, I think, is the clear message to the team: “if the system crashes, you’ll be the ones to be called out of your bed at 2am”.

    Filling in the details then becomes a mere matter of cooperation, experimentation, evaluation, and reliance on the collective intelligence of the people in the teams. As we all know, you’ll always be surprised by the latter, so prescribing too much too early would be, IMNSHO, unproductive.

  6. Aliaksei Yanchuk says:

    I haven’t made it to the meetup, so I cannot comment on the presentation itself. Maybe you have slides?

    All these thought are nice, but IMHO there are two more columns missing from your table: the users of the system / product and the sponsors of the product. Users want to get their work done — whatever they understand as work, as well as under getting it done. Sponsors usually want profits, be it financial, political, personal — that all depends. Leave these two groups out of your picture, and you most likely will end up with development without knowledge and operation without a clue. šŸ˜‰

    I don’t think that there is a divide of any kind between Ops and Dev as long as two parties can listen to each other and truly understand and adapt to each other’s challenges. Sticking to the letter within organisation wouldn’t help anyone, it is a form of strike. Can you imagine Scrum to the letter in Ops? I.e. we will do only X installations this week, and no installation more or less? Even if half of the organisation is sitting without underpinning services? šŸ˜‰

    There is divide of another kind: prohibition of big design up-front. Although the benefits of this approach are clear, it must be used with caution. Developers are humans first, and humans tends to delve into areas where they feel themselves most convenient — into development, without considering the complexity of the project ecosystem around it. It may turn out that your project will be “illusion-led” until Ops ban it for production.

    In order to arrive to a working system (do not confuse it with passing tests) that makes everyone happy, you need to be at least clear about your long-term goals. E.g. how much scalability will you need in the end? Are you developing a system for 20 users or for 20K users? Where are the quick wins, and where are long-term benefits? How would these be delivered? If the project fails to nail your coordinate system down (hellow BDUF!), all nice metrics won’t be telling you much of useful information.

    Here is your mindset change.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s