Disclaimer: I came to Pocket as a very seasoned application developer but essentially a novice/reluctant dev-ops/SRE/infra developer. This context will be demonstrated below via naïveté and snarky humor. Please enjoy.

In the beginning, there was CloudFormation

Webster’s, ahem, I mean, AWS defines CloudFormation as:

“…an easy way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycles, by treating infrastructure as code.” ¹

Somebody bring some electrolytes and run an ice bath because “easy” is doing A LOT of work in that sentence. JSON and YAML are easy enough to write [2], until you get past some fuzzy threshold (which I’ve never seen not crossed in a real-world scenario), at which point trying to comprehend a full infrastructure stack is on par with putting together a 1000 piece jigsaw puzzle consisting only of letters and odd punctuation.

As our infrastructure evolved (meaning more tinkering with our CloudFormation templates), and as new developers were brought on (hi, it me), it became very clear that CloudFormation was not the best way forward for us. (Real talk, the pre-existing CloudFormation was legacy - I don’t believe anyone liked it. Time and priorities - they are things.)

Have we been Terraformally introduced?

When I arrived at Pocket, the majority of the infrastructure was written in Terraform using HCL (HashiCorp Configuration Language). (Is saying “using HCL” redundant here? Does anyone write it in JSON? I sure hope not.) Coming from CloudFormation, HCL was a huge breath of fresh air. Logical blocks! Separate files!² Variable referencing!³ Okay, that last one isn’t a great example of a benefit, but it’s hard to argue that HCL isn’t far easier to write, read, and comprehend than CloudFormation.

If you’re following along in the footnotes below (and honestly, it’s probably not worth your time, I just enjoy writing this way), or if you’ve experience writing and managing HCL, you’ll know it’s got its own set of issues. The levels of abstraction are nicely simple, but tend to become sprawling and nebulous once your HCL gets to any significant size. Custom modules and arbitrary-ish separation of concerns through file structure can make coming in/back to the code rather challenging. The complexity feels flattened - wide and shallow, as though I need to be aware of all (okay, much) of the HCL to understand a small chunk. HCL kind of feels like Post-It notes - a great way to manage a small set of information, but not great at scale.

I will take HCL over CloudFormation 100% of the time, but I will still be left looking for something better.

Lest it go unsaid, and this is a HUGE fly in the ointment of infrastructure as code, is that any configuration files - be they YAML or HCL or JSON - need to exist for each application. So what do you do? You fix up a whole buffet of copy pasta, duplicating your configuration into each new application, tweaking as necessary. And what happens when you need to make organization-wide changes to your infrastructure? The DRY goddess weeps, and you weep with her. (This is far easier in HCL than CloudFormation, at least.)

A TypeScript Side Quest

The details are for another post at another time, but as we came to spin up new micro services, our team was faced with a decision - what language to choose? Because reasons, we settled on JavaScript - most well known (on average) on our team, lots of community tooling, is everywhere these days - but we also wanted a bit more assurance than vanilla JS can give. TypeScript was the obvious answer. After using TypeScript on a few small projects, we quickly developed a pattern/flow with the language. Like skipping documentation, it came easy to the team. (I kid! I kid! ⁴)

(I’ve been writing JavaScript for a very long time and TypeScript is the first “extension” that feels natural. CoffeeScript was junk I will hear no other opinions on the matter good day.)

CDK - Terraform’s Interfacelift

It was with hushed excitement that the team first read about the Terraform CDK. Nervous, anxious glances were exchanged (you can exchange a glance over Slack, right?). Could it be? Can we have interfaces and return types and classes and functions to manage our infrastructure code? Beads of sweat pillowed on our foreheads. Were we ready for such an upgrade? Could it be the holy grail we had been waiting for? ⁵

It took only a proof of concept for everyone to get on board. We had a type-safe language ⁶ we were all familiar with working against a known infrastructure provider’s (HashiCorp) development kit that could encapsulate functionality and provide predictable and configurable return values. What had we done to deserve such kindness?

And yes, because it’s just classes and functions, it’s also testable. Excuse me for a moment while I dance in the mirror.

But this wasn’t even the best part. The best part is that we were able to publish our specialized, abstracted infrastructure library as an NPM package. The days of cooking copy pasta were soon to be behind us. Instead, we can feast on actual application code, sending only the necessary configuration options to our abstracted infrastructure library. No more code duplication. And. AND! If we need to make a sweeping, organization-wide change to our infrastructure? We change code in ONE place. ONE repo. All that’s left is to update the NPM dependency in our other projects, and the change is inherited.

BRB, I need to go dance again.

post.synth()

Perhaps it’s the application developer in me talking, but I personally don’t think YAML or JSON or HCL are sufficiently sophisticated enough to handle infrastructure as code ¹. Not at any kind of complexity and/or scale, anyway. Encapsulation, reusability, readability - these things are necessary when writing any kind of application, of which infrastructure is just another example. As of now, we’re still early in the process of abstracting our existing copy pasta HCL into an abstracted and reusable TypeScript library. But we’ve got a couple applications running on it, and I personally am so grateful to not be spending half (if not more) of my time writing and testing copy pasted infrastructure code.

Oh, humble brag, we recently open-sourced our terraform-modules repository. If you’re interested, take a look. (It’s probably way specific to how we do things here at Pocket, but at minimum could provide a nice demo for rolling your own.)

Good luck to all of you application developers turned dev-ops coders. You have my respect and sympathies.

Until next time, take care of each other, and I’ll see you on the internet.

~ jonathan, backend team

1. Someday I want to explore the impact of “infrastructure as code”, but alas this is not that day. ⁷ ↩ ↩²

2. But it’s up to you to make sure your file structure makes sense! And there aren’t any standards that I’ve seen! Oh no! ↩

3. The block is called local but the variables are referenced using locals. I would love to know the backstory as to how this decision was made. ↩

4. Kinda. ↩

5. Spoiler alert - there is no holy grail in technology. Only situationally appropriate decisions, and situations rarely stay static. This applies to life, too. ↩

6. Terraform’s CDK also supports Python (yay!) and Java (oookay?), so if you’re too good for JavaScript (or in the case of Java, not good enough), you’ve got options. ↩

7. I also want to explore the pros and cons of keeping infrastructure in the same repo as the app code, but this is also not that day. ↩

Tagged with: #infrastructure, #terraform, #typescript

---