No more painful Lambda Local development: Unboxing the SST Magic!

Data Eng Weekly - Ep 20

This week, I had a look at a new serverless framework called SST (Serverless Stack Toolkit).

Primarily aimed at building full-stack web applications on AWS, SST is particularly effective for constructing serverless data applications as well.

The cloud development experience offered by SST is very good: it simplifies and streamlines the entire lambda development process, making it a pleasure to work with compared to other frameworks such as SAM, Serverless, or CDK.

I highly recommend giving it a try for your next project.

Live Lambda Development

What truly captivated me about SST is its capability to execute Lambda functions locally within the AWS environment. This feature not only simplifies development but also significantly enhances the testing process.

Live Lambda Development or Live Lambda is feature of SST that allows you to debug and test your Lambda functions locally, while being invoked remotely by resources in AWS.

Indeed, working with Lambda can be quite challenging, especially when it involves interacting with other AWS services.

Before SST, developers had the following options to test their lambda functions:

1. Code → Push → Wait for CI to finish … → Test in AWS console.

   This process can be quite slow and cumbersome.

2. Mock external services.

   This can work but requires a considerable amount of work and maintenance.

3. Use LocalStack, a service that emulates AWS locally.

   Although this option can be viable, it can also be difficult to maintain and doesn't handle complex use cases well.

SST, however, has emerged as an efficient alternative to these traditional methods.

SST resolves these challenges by creating a temporary Lambda function in AWS, which routes the invocation events to your local Lambda function.

This means you can effectively work on your Lambda functions locally, while still interacting with actual AWS services.

From the SST doc:

1. When you run sst dev, it deploys your app and replaces the Lambda functions with a stub version.

2. It also starts up a local WebSocket client and connects to your AWS accounts' IoT endpoint.

3. Now, when a Lambda function in your app is invoked, it publishes an event, where the payload is the Lambda function request.

4. Your local WebSocket client receives this event. It publishes an event acknowledging that it received the request.

5. Next, it runs the local version of the function and publishes an event with the function response as the payload. The local version is run as a Node.js Worker.

6. Finally, the stub Lambda function receives the event and responds with the payload.

In development mode, SST lets you make updates to your Lambda handler in Python locally and test changes directly without needing to deploy each time.

Even better, you're interacting with actual AWS services, which means you avoid running into access rights issues after deployment.

The lambda running locally, you can therefore easily link it to your VS Code debugger !!

This live testing environment makes for an incredibly smooth and efficient development process.

Let’s have a look at an example

SST is built on top of AWS CDK and comes with a variety of built-in constructs. These pre-packaged modules can make it much easier and faster to deploy common architectural patterns and functionalities.

If you need more specific or complex AWS services or configurations that are not covered by SST constructs, you can indeed fall back to AWS CDK.

This is one of the major strengths of the SST framework, allowing developers to work effectively at both high and low levels of abstraction.

It gives you the simplicity of SST constructs for common use cases, but also the power and flexibility of AWS CDK when you need to get more hands-on with the underlying AWS infrastructure.

Let's go through an example of using SST.

Here, we'll create a simple Lambda function that writes data to an S3 bucket:

It’s interesting to note that the lambda is bound to the s3 bucket with the property: bind:[bucket].

The bind feature in SST indeed simplifies the process of setting up communication between AWS resources.

In the case of binding a Lambda function to an S3 bucket, SST automatically handles the creation of necessary IAM roles and policies for accessing the bucket!

Moreover, it also sets the bucket's name as an environment variable within the Lambda function.

I wrote the following lambda handler in Python which simply upload a Pandas dataframe to S3:

And after running

npm sst dev

SST created the bucket S3 in the AWS account and run the lambda handler locally.

SST provides as well a web-based user interface called SST Console that allows you to manage and monitor your applications easily.

After invoking the Lambda from this console I got the CSV file created in the AWS S3 bucket as expected.

To deploy your stack, you can simply then run the following command:

npm sst deploy

and a Docker-based Lambda is deployed to AWS.

That’s all I wanted to present about SST.

For anyone who's interested in experimenting with serverless architectures, it seems like SST provides a great user experience, particularly for those working with AWS Lambda functions.

Feel free to share your thoughts and experiences as well!

---

Thanks for reading,

-Ju

I would be grateful if you could help me to improve this newsletter. Don’t hesitate to share with me what you liked/disliked and the topic you would like to be tackled.

P.S. you can reply to this email; it will get to me.