paulmacp's blog

The internet is an interesting medium in which to work. Almost all forms of business are now being conducted on the Internet, and people continue to capitalize on the fact that a global audience is on the other side of this virtual drive-thru window.

The Internet changes so quickly that we rarely have time to truly understand the impact of what we are doing and why we are doing it. When it comes to operating the fabric of an online architecture, things move so fast and change so significantly from quarter to quarter that we struggle to stay in the game, let alone ahead of it. This high-stress, over stimulating environment leads to treating the efforts therein as a job without the concept of a career.

What's the difference, you ask? A career is an occupation taken on for a significant portion of one's life, with opportunities for progress. A job is a paid position of regular employment. In other words, a job is just a job. (but Web Operations is a pretty good job as far as jobs go.)

Although the Internet has been around for more forty years, the Web in its current form is still painfully young and is less than twenty two years (since HTML first conceptualized at CERN in 1990). I myself first learned HTML in 1995. So, how can a person fill a significant portion of one’s life with a trade that has existed for only a fraction of the time that one typically works in a life-time? At this point, to have finished a successful career in web operations, you must have been pursuing this art for longer than it has existed. In the end, it is the pursuit that matters. But make no mistake: my pursuit of a career in web operations makes me a (technology) frontiersman.

The field of web operations is one with which I am intimately familiar. For the last 15 years, I have immersed myself in this emerging field (from my Years at Sun Micro Systems, Long View Systems and Now TVO). Even now, writing a job description for a web operations specialist is nearly impossible and when I speak with colleagues about what web operations truly is, we all seem to articulate things differently.

Over the past few years I have realized two things:

1. a realization of the lack of a career path for people who do what I do (no standard titles, no standard roles and responsibilities);
2. a clear lack of terminology for the technology requirements that are so common in these environments.

Terminology is easy, in my opinion — you just argue until someone wins. Of course, arguing is a hobby of mine, so I have a bias. On the other hand, defining a career path that is an industry accepted path is hard.

The Job: Web Operations

The captains, superstars, or heroes who live in the Web Operations roles are multidisciplinary experts. They have a deep understanding of networks, routing, switching, firewalls, load-balancing, high availability, disaster recovery, TCP & UDP services, NOC management, hardware specifications, several different flavors of UNIX, several web server technologies, caching technologies, several databases, storage infrastructure, cryptography, algorithms, trending and capacity planning. The issue: how can we expect to find good candidates that have fluency in such a multitude of technologies? In the traditional enterprise, you have architects which are broad and shallow and their team of experts which are focused and deep. However, the expectation is that your “web operations” engineer be both broad and deep: fix your gigabit switch, optimize your MySQL database and guide the overall architecture design to meet scalability requirements.

I struggle with this. Not everyone can be a superstar. More importantly, no one can really start as a superstar. So, how do we define the requirements for a junior web operations person?

We have to have a plan for hiring on people and progressing them through a career path to make this a legitimate discipline. During conversation, one of my colleagues said they just hire people that they think are agile — “If I tell them to know IOS well enough to configure a router and troubleshoot a problem, I expect them to show up tomorrow with a basic understanding of IOS and ready to start typing in commands at a console.” I agree this sort of “no boundaries” attitude is required for the job, but where do you start?

Basically, this is one of the few positions in the organization that has no boundaries of responsibility. If something breaks, it is your problem. Why isn’t this the case throughout the organization — why is it that even the most junior of developers doesn’t wake up to fix their code when it breaks and causes service degradation in the middle of the night? It is uncommon that this level of responsibility is expected of developers, while it is a quite common expectation of the operations crew.

Circling back, I really do not like the term “web ops.” I realize it is not far off, but it isn’t sexy. Google has a few different roles with this level of responsibility. One I like is called: “Site Reliability Engineer.” However, I would like a set of job titles and a progression through them that makes this an appealing career path for young, ambitious geeks.

In order to define these roles, we should think about what they are responsible for. In our organization I see this as a few things:

Junior

On the junior level, they are responsible for learning. They are responsible for deploying new services and documenting such deployments. They are responsible for instrumenting deployments to make sure that faults are detected and trending is possible.

Mid-level

On the mid-level, they are responsible for all of the above, and more. Effective and complete troubleshooting of failures. Making sense of trending information. Understanding workloads that exist. Tuning systems to better accommodate current workloads and proactive tuning to handle known future workloads. One of the key differences between mid-level and junior is the ability to correctly prioritize remediation of issues during incident response. Staying calm, collected and executing with clarity of thought during an emergency.

What does “complete troubleshooting” mean? I mean troubleshooting without boundaries. I want no shyness in cracking open developer code and telling them what they did wrong and why. Finger pointing at people simply doesn’t work, you have to point your finger at implementation problems, not people. To do that requires the skill to track a performance problem or reliability issue down to a specific line of code or approach.

Senior

On the senior side, technology research and selection is a must. Additionally, they are responsible for incorporating new technologies in the architecture to improve availability and reduce costs, constantly analyzing systems to improve efficiency and capacity planning to understand growth well enough to ensure provisioning and deployment outpace need.

One of the core responsibilities that all engineering disciplines share is assessing the appropriateness of the technologies at hand. For example, a “Web Architect” must ensure that technology selection as well as development and deployment strategy match the business need. This is “hard.”

Above and Beyond

Web operations is a special role. It is for developers/engineers that have outpaced their career path. One that has a deep understanding of how things work: “a complete systemic view of general site architecture.” However, they want more responsibility, they want to make sure that all of it works all of the time: the app, the stack, the hardware, the network. Whatever technology the business needs, it must work, it must perform and it must be able to meet demand. Lastly, in their heart of hearts, they must believe that all problems are equal in their need for resolution and problem prioritization is dictated by business impact and not by flights of fancy (how cool or interesting the problem is).

It is an impossible job requirement: “Knows everything about all technologies deployed in Internet architectures.” While I certainly don’t fill this requirement, no one does. Someone who wants to pursue a job in Web Operations is someone whose career goal is to find out how close they can get.

[This is a repost of somehting I put together for TVO's (my employers) internal portal]