MiniAppEngine (Building a PaaS from scratch)
I normally don't like talking, much less blogging, about projects I haven't finished. However, through my conversations with friends and colleagues I've realized that I've stumbled upon something interesting both as a project and as a process.
I discovered Google App Engine through a Udacity course on Web development taught by Steve Huffman back in 2014 I believe. I was amazed at how easily I could deploy my apps through a CLI. Ever since then I've always had a soft spot for the simplicity that a Platform-as-a-Service (PaaS) like App Engine can provide. I even got to use App Engine for a freelance project which was a lot of fun.
Fast forward to November 2023 and I wanted to take on a project that would challenge me. I also wanted to use ChatGPT to help me take on projects that would have intimidated me before. Remembering my experience with Google App Engine I decided to build a simple version of a PaaS aptly named MiniAppEngine. Obviously, this cannot be a 1:1 clone of App Engine so I decided to focus on the following objectives:
Only deploy Python apps (like the first version of App Engine)
Deploy through a CLI
Auto-scaling to 2 instances and load-balancing
Learn OS, networking, containerization, and orchestration fundamentals
Learn software engineering best practices to achieve modularity
I decided to use TypeScript to build the CLI, Ruby on Rails to build the web API and UI, and Elixir for the control plane. Eventually the control plane would be replaced with Kubernetes but starting with Elixir will help demystify how orchestration works.
In order to achieve modularity I decided to follow Hexagonal Architecture and EBI. Hexagonal architecture primarily focuses on interactions between modules which is why this pattern is referred to as "ports and adapters". Each module exposes its business logic through ports so that other modules can connect to it with adapters. This way the internals of a module are not susceptible to change when something outside it changes. For example, the deployment module will need to create a background job for the control plane. The deployment module could directly pass the job to RabbitMQ. However if tomorrow I decide to replace RabbitMQ with Kafka, I'd have to change the module as well. Instead the deployment module could pass the job to the background queue through a port and RabbitMQ can access it through an adapter. Now, swapping out RabbitMQ with Kafka should localize changes and not affect other modules.
Structuring modules likes this enforces contract-driven-development. Each module maintains a contract which other modules have to abide by in order to interact with it. As long as the contract doesn't change, changing the internals of a module should not affect other components and modules. This sets the stage for extracting modules into microservices in the future if such a need arises.
Another benefit of enforcing modularity is that theoretically the application can be framework-agnostic. In this case then for example, the business logic of the web API could theoretically be ported to any Ruby framework like Sinatra or Hanami as gems. Furthermore, the business logic does not need the entire Ruby on Rails framework for running its tests. The web framework then becomes just a delivery mechanism, i.e. a technical/ implementation detail. In other words, Rails is not my application.
While Hexagonal architecture is great for designing how modules should interact with each other, it doesn't say much about how modules should be structured. This is where EBI can help and I learned about it from a talk by Uncle Bob. Each module can have entities which represent the objects in the module, boundaries which expose the entities to the outside world, and interactors which contain the business logic that manipulate the entities.
One final benefit that modularity provides which makes this project even more exciting is that once the business logic is contained in gems, I could iteratively port the code to Rust. Bundler supports Rust code in Ruby gems so this should be fairly straightforward. Doing so would allow me to convert the business logic into WASM which means it would be language-agnostic. Theoretically, I should be able to port it to Django, or NestJS, or Spring Boot...
As you can see, I'm very excited to work on this :)