Welcome to the exciting world of data engineering!
Every week, you'll get a chance to learn from real-world engineering as you delve into the fascinating world of building and maintaining data platforms.
From the technical challenges of design and implementation to the business considerations of working with clients, you'll get a behind-the-scenes look at what it takes to be a successful data engineer. So sit back, relax, and get ready to be inspired as we explore the world of data together!
If you're working with cloud-based software, you're probably creating, deleting, and updating various software objects like S3 buckets, databases, user roles, and more.
Traditionally, managing these assets was done through the cloud console, which could be a tedious and error-prone process.
But with Infrastructure as Code (IaC), you can manage these assets with code instead. This means that the creation, deletion, and update of your cloud objects are managed through code rather than manual configuration through the console.
Managing these assets with code has the following advantages:
Let’s dive into the implementation of IaC.
There are multiple frameworks to choose from, each suited for different contexts.
Check out the image below for some popular options used to manage AWS cloud infrastructure.
Different criteria can be used to distinguish between IaC frameworks:
Open source. Some frameworks are released by AWS directly.
Versatility
Terraform for example can be plugged into almost any cloud software. There is a wide range of so-called “provider” that allows you to manage almost any possible software: Gitlab, Github, Snowflake, AWS, Azure, GCP etc.
Other frameworks are cloud-specific and built as a layer on top of these low-level frameworks. SST, CDK, SAM, and Serverless are for example built on top of Cloud formation which is a native AWS service.
Abstraction
In the AWS context, a way to understand the difference between the existing frameworks is the concept of abstraction
To better understand abstraction in the context of IaC, let's compare Terraform and Serverless Framework in a concrete example.
As shown in the image, Terraform requires you to declare all the objects that make up your infrastructure: a bucket and a lambda. Additionally, like in the console, you must add authorization and roles to these objects.
This process can be tedious and verbose - for this simple pattern, you need to write 85 lines of configuration code!
This is where abstraction comes into play.
Let's have a look at its equivalent to Serverless:
Serverless can do the same job with just 15 lines of configuration code.
The framework takes care of resource binding: it creates for you all the required permissions.
The trade-off is simple: you may lose some control over your infrastructure, but you can speed up development.
Another typical use case is the creation of an API.
In serverless you just have to declare this code to create an API endpoint triggering a lambda function:
# serverless.yml
functions:
index:
handler: handler.hello
events:
- http: GET hello
In Terraform you would need to declare the following resources (~90 lines of code):
aws_lambda_function: lambda function
aws_iam_role: role of the lambda
aws_api_gateway_rest_api
aws_api_gateway_resource
aws_api_gateway_method
aws_api_gateway_integration
aws_api_gateway_deployment
aws_lambda_permission
We can clearly see the added value of abstraction to help you gain speed and efficiency.
I personally use the following setup in my project:
Terraform is ideal for low-level infrastructure where you need a lot of control: VPCs, Permission boundaries, admin roles.
For building applications themselves, I prefer to use Serverless or SST for speed and faster development.
I hope this post helped you understand the added value of IaC and why it's an essential skill for anyone working on the cloud.
thank you for reading.
-Ju
I would be grateful if you could help me to improve this newsletter. Don’t hesitate to share with me what you liked/disliked and the topic you would like to be tackled.
P.S. you can reply to this email; it will get to me.