JA

How to build real technical skills in system operations and maintenance, the dumping ground for useless new grads

Here is what this post covers:

Introduction

Did you know that software development has an operations phase that comes after the development phase? When people hear “operations and maintenance,” most of them picture a place to park useless new grads who barely understand anything technical, or a team that cleans up the bugs developers missed in testing, runs routine tasks, and maintains data received from other systems. In short, garbage work. In fact, I did ops and maintenance myself back when I was a useless new grad without the faintest idea of what technical skill even meant, and I never felt like I was getting any better.

Even in a situation like that, could I have found ways to build my skills? I want to look back and think about it.

In this post, I first describe what my job looked like as a new grad. Then I think about what I would do if I could start over, and what you can only really learn in ops and maintenance.

What I did as a new grad

The piece of garbage with no technical skills (me) was in charge of running and maintaining servers such as DB servers and batch servers. These servers ran jobs that took in data sent over from other systems, transformed it, and exposed it as views so analysts could use it.

The work was completely different between day and night. During the day, it was mostly routine tasks done by following a runbook. At night, if a batch job fell over, the data would not load correctly, so I fixed the data itself.

During the day, following the runbook got the work done. At night, fixing the data made it ready in time for the next morning’s analysis. Both were jobs you could get through without really understanding how anything worked.

If I could start over

Looking back now, there was room to build skills even within the same work. If I could go back to being a new grad, I would do these four things.

Look up the “why” behind the runbook

A runbook tells you which command to type or which SQL to run, but it often does not say why. So I would go through it line by line and look up what each command does, and what would happen if I swapped the order of the steps.

Even with Linux commands you use every day, looking up the options one by one teaches you a lot about the OS and how it handles files. If you add what you find to the runbook, it helps whoever takes over after you, too.

Turn routine work into scripts

If you repeat the same steps from the runbook every day, those steps can very likely be bundled into a shell script or a SQL script. To automate them, you have to understand what each step does and where it can fail. Writing the script is itself practice in understanding the procedure.

Automating something that rewrites data right away is scary, so I would start with steps that do not change data, such as checks. Some workplaces require approval before you run your own script in production, so talking to your manager and team is a must.

Track down why the batch failed

When handling failures at night, my goal was to fix the data in time for the next morning. If I could do it over, after fixing the data I would read the logs and pin down which job failed on which data.

Did the upstream system send a value nobody expected? Did the data volume grow until the job could not finish in time? Was a condition missing from the SQL? Once you know the cause, you can bring the development team a concrete fix. If the same failure keeps happening, that is also evidence that the processing should be fixed, not the data.

Draw a diagram of how the data flows

I would draw, myself, the flow of data from the upstream systems, through the batch jobs that transform it, to the views the analysts use. Which job reads which table, and which table does it write to? Which job has to finish before the next one can start?

With that in your head, when a batch fails at night you can quickly judge how far downstream the damage spreads. Drawing the diagram means reading table definitions and job settings, so you also pick up knowledge of databases and job scheduling.

What ops and maintenance teaches you

So far I have looked at how to build skills inside ops and maintenance work. Flip it around, and there are things ops teaches you that are hard to learn from development alone.

Production data and incidents

Test data used during development tends to be clean, but production data is not. Different character encodings, unexpected NULLs, numeric overflows, duplicate records: problems that never showed up in testing happen in production.

Ops and maintenance puts you in front of these problems every day. Once you know how data breaks, you can judge how much input validation and error handling to write when you are the one building the processing.

Seeing the whole system

In development, you can sometimes get the job done by knowing only the feature you own, in depth. In ops and maintenance, on the other hand, you cannot isolate the cause of an incident unless you look at the whole path from the upstream systems to the analysts.

Hunting for causes across layers such as servers, databases, batch jobs, and the network carries over directly when you design systems.

A sense for what is easy to operate

If rerunning a failed batch does not produce the right result, the only option is to fix the data by hand. Whether a rerun can recover things depends on whether the batch is idempotent. Idempotent means running it any number of times gives the same result. With an idempotent batch, recovering from a failure is just a matter of removing the cause and running it again.

Beyond that, you get a feel for things like what to write to the logs so causes are easy to trace, and where to split the processing so it can restart from the middle. That feel comes more easily when you have struggled with operations. It helps you build systems that are easy to operate once you are on the development side.

Experience that leads to SRE

Google uses the word “toil” for the kind of work tied to running a production service that is manual, repetitive, automatable, has no enduring value, and grows linearly as the service grows. Google’s SRE (Site Reliability Engineering) teams aim to keep toil below 50% of each person’s working time. The rest goes to engineering, such as work that reduces future toil and adding features.

The runbook-driven routine work I did as a new grad was toil, pure and simple. Seen another way, an ops and maintenance team is a place with a mountain of toil right in front of you, waiting to be reduced. If you build experience finding toil and automating it, you can move toward work like SRE, which improves operations through engineering.

Wrap-up

Ops and maintenance is mostly routine work from a runbook and cleaning up after failed batches. If you just get through it without thinking, it is a job where your skills barely grow. As a new grad, I also got through each day’s work without ever feeling my skills grow.

But looking back, the same work had plenty of material for building skills. Look up what the runbook means, turn routine work into scripts, track down why batches failed, draw how the data flows. All of these start with digging into why the work in front of you is done the way it is.

Getting to touch production data and incidents, seeing the whole system, and learning what makes a system easy to operate are experiences unique to ops and maintenance. Even if you are assigned to ops and maintenance, you can build real technical skills. Whether you stop at getting the work done, or dig into the reasons and become the one who changes how things work, makes a big difference in what you take away, even on the same team.

References