At a recent meeting, I was told such-or-such changes would be postponed, because we simply didn’t have enough DevOps at the moment. I innocently retorted “I thought DevOps was a practice, not people!”. The answer to that will surprise you!

“True, but we’re not quite there yet.”

This got me wondering. What is DevOps?

The Wikipedia article is quite bad. Its biggest section is “relationship to other approaches”. Talk about an identity crisis. “Toolchains” is a top-level section. Is that because it has specific discriminating tooling? “Goals” is a top-level section too, and it’s much bigger than “Definition”. It reads more like “Dreams” anyway.

And the money quote near the end:

While DevOps describes an approach to work rather than a distinct role, job advertisements are increasingly using terms like “DevOps Engineer”.

Come to think of it, yes, I too have had to recruit DevOps contractors before. I distinctly remember asking, at the time, what was meant by that. “Well, someone like us, you know, understands his shit and is able to operate less stiffly than the operations’ silo.”

“You know”

At least I was able to filter out those who put bash on their resume just because someone had told them to.

And by evaluating the line-up, I was bestowed with part of the revelation. The DevOps candidates I was presented were pieces of meat from “lesser” schools, not because they cost less, you pessimist, but because, you know, it’s a specific skillset, that happens to be sold to us for more because, you know, they’re kind of hard to come by, and from a distinct set of suppliers, because, you know, that’s the DevOps way.

Money finds its way.

Speaking of money, let’s peek at Microsoft’s take:

A compound of development (Dev) and operations (Ops), DevOps is the union of people, process, and technology to continually provide value to customers.

What does DevOps mean for teams? DevOps enables formerly siloed roles—development, IT operations, quality engineering, and security—to coordinate and collaborate to produce better, more reliable products. By adopting a DevOps culture along with DevOps practices and tools, teams gain the ability to better respond to customer needs, increase confidence in the applications they build, and achieve business goals faster.

Cue marketing pitch, along the lines of “teams that adopt DevOps culture, practices and tools become high-performing, building better products faster for greater customer satisfaction.” Better than your local astrologer or your money back!

The sharp reader will have noticed that despite leading the sentence with “formerly siloed”, nowhere does it say the teams’ structure would actually change.

Let’s try another big name, such as AWS.

DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. This speed enables organizations to better serve their customers and compete more effectively in the market.

Well at least now it reads like a definition. It’s the set of stuff that lets you deliver fast. If you deliver fast, you’re DevOps.

My team was delivering faster than the global organization’s average,¹ so at least part of that could have rung true, had I sought enlightenment at the time.

Let’s try the open-source world, with RedHat’s view on the matter.

The word “DevOps” is a mashup of “development’ and”operations” but it represents a set of ideas and practices much larger than those two terms alone, or together. DevOps includes security, collaborative ways of working, data analytics, and many other things. But what is it?

Pray tell.

I’ll add Atlassian’s view for good measure. Because it was on the search results’ first page, not because I blindly trust their unbiased opinion on the matter.

DevOps is a set of practices that works to automate and integrate the processes between software development and IT teams, so they can build, test, and release software faster and more reliably.

Well at least it’s humble. The rest of the page peddles their software suite as various “stages of DevOps”, so I’ll snip right there.

Yet they do it with that fine infinity-shaped loop diagram. The one all of the others used as well. Maybe there’s something to be found there? Here’s one from Wikimedia Commons.

It’s the first one I see that actually makes sense to display as two loops, since it splits parts of the process between Dev and Ops. The big names above just had the seven phases, mapped to their products.

But hey, at least with this chart, we can completely split the reponsibility between Dev and Ops, crystal clear as day. It’s made very close to explicit by the final link on the search page, The Agile Admin.

DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.

DevOps is also characterized by operations staff making use many of the same techniques as developers for their systems work.

At last, it’s “official”! DevOps can’t be a role, since the chart and the Agile Admin mention dedicated developers and operation staff.²

I have no idea what those people I recruited were, then.

I’m not out of options just yet, though. DevOps is taught in schools, now. I’ll simply ask a trainee what it’s all about.

“DevOps? Oh, I know that one, we studied it last semester! It’s, err, GitHub and Jenkins, right?”

You tell me!

It only makes sense. It sounds cool. So branding wants their share. Before you know it, everybody is using it. So everybody has to have it as a listed resume skill. So universities have to provide, and are happy to oblige. Whoever came up with that name is a marketing genius.

Hey, that’s a lead. Who coined the term?

Digging that up is harder. But the French Wikipédia article has an answer: Belgian Patrick Debois did. So I’ll assume he’s the French-speaking flavor of belgian, and community spirit ensured he was properly listed in all the relevant places.

Following the link to his entry, we get a proper explanation from the horse’s mouth: he used the term to organize the first devopsdays in Ghent, because “Agile Administration System” was too long.

Agile Administration System

At last!

I’ve also read about Agile Infrastructure, which hits closer to home for me. Hey, I’ve worked with organizations where the hardware to run their applications had to be planned two years in advance if you wanted a fighting chance at having it in time.

Bringing the wonders of agility to infrastructure and deployments! Indeed, using continuous integration and proper version control can only be a step up from where operations used to be. Continuous deployment based on a good implementation of infrastructure-as-code could definitely help too.

This reminds me of what a few friends of mine do at Google. What did they say they called it? Ah, yes, SRE.

For those who’ve been out of the computer industry for the last two decades, Site Reliability Engineering is Google slang for “automate the shit out of anything that’s in the way of www.google.com having five nines of uptime.” I may have gotten some details wrong, I could only get the googlers to talk to me after getting them drunk and playing pretend you’re having your hiring interview again. Secretive bunch.

In contrast to DevOps, SRE appears to be a very explicit role. And organization. It’s kind of funny everybody else has SRE positions open too, given the little power they have to actually keep www.google.com up. Or down.

I jest. “S” can stand for whatever your TLD is. It even extends to systems nobody in their right mind would call a site.

Everybody wants to have them because the concept comes from Google, and when they’re not sunsetting Yahoo, everything Google does is cool. So branding wants their share of being cool too. Before you know it, everybody is doing it. So everybody has to have it as a listed resume skill. For some reason, universities don’t seem to provide; they’re still busy churning out DevOps.

I figure there’s only two kinds of SRE job openings out there.

1. the ones from Google.³
2. the ones from companies who want to become Google by walking their footsteps; but in the meantime you’ll just operate the software their developers throw over the fence whilst refraining from distracting them. At five nines. You’ll fail on both counts.

What a mess!

Not being Google doesn’t preclude you from automating the hell out of a lot of shit, though. God knows my team and I have automated⁴ a lot over the years.

Take Hudson, for example. It wasn’t always there.⁵ Countless in-grown alternatives, including two of mine, were proudly triggering the industry’s software builds in response to developer commits. Striving to minimize duplicate work in a cross-platform sea of C++ ABI compatibility brain teasers. Then to build on top of that to minimize developers’ build times.

Then to have the hardest time convincing management we should migrate to Jenkins, because its impetus dwarfs our sunk costs by orders of magnitude. Some things never change.

Or that other DevOps tool, “GitHub”.

By pure luck, I happened to be on the lookout for version control software when git first came out. So I happen to use it since week one. Stating it was different then would qualify as understatement of the year. Like, there’s-no-commit-command level of different. That’s how I got my knowledge of the git internals since the beginning: back then, there wasn’t anything but. At least it gives me the confidence to do crazy stuff with git dependency graphs and have reddit melt my webserver in return. I can also do the normal stuff, now git’s got the normal stuff abilities too.

The organization I contracted for at the time was seeking justifications to spend manyears on in-grown version control, so they launched a survey of existing options. I entered one-month-old git in, semi-jokingly, clearly listing its young age as a lack of maturity con.

Naturally, it got rejected. Not for lack of maturity, though. I had listed its distributed abilities as a pro, so…

“It’s distributed whereas we require a centralized system.”

Wow.

Of course they too are using git by now. It’s still as distributed as it was back then. It’s even better at being distributed now than it was back then. They’re using it centralized anyway. As was possible back then as well. Some things never change.

What’s left in the DevOps standard toolkit? Chef and Ansible? Nagios and ELK? LXC and Docker? Kubernetes? I’ve hacked on in-grown alternatives, around proprietary ones, and migrated systems to them.

What’s left in the SRE standard mindset? Automating all that can? Engineering continuous-availability releases? Adjusting alerting to keep on-call rotations meaningful and humane? I’ve done plenty of those too. Shortening the release cycle? Sounds more like DevOps, but it’s a process, and I’ve done a fair share of it too.

Does that make me a DevOps-abiding SRE? Answering that would warrant another post. I do think I’ve got the minimal credibility to apply to such a position, though.

“Hello, our standard recruitment pipeline would normally have you pass a test to see whether you qualify as an SRE, but in your case we’re going to skip it altogether: you have too much out-of-domain knowledge for us to consider you can possibly have the advanced skillset we require for this position.” ⁶

Some things never change.⁷ 🙃