Today, after more than five years of dedication, the future of DevOps automation is here. System Initiative is now generally available for everyone to use.
Three Years to Three Clicks Link to heading
About three years ago, Adam took me through a live demo of the System Initiative (“SI”). I wrote about how transformative that experience was when we launched the company publicly, but here’s the short version: I dropped everything in the weeks following that demo in order to join as the second engineer and fifth employee of the company.
The following three years have been some of the most rewarding in my career. Our world class team, and steadily growing and passionate community, have been vital to not only reaching this milestone, but also to my own career. I cannot thank them enough.
You may or may not have heard of us over the last year or so, but in any case, let’s talk about what’s different now.
Last year, we launched the company publicly, open sourced the System Initiative software and steadily built an excellent community of people looking for the future of DevOps in our Discord server. However, all of these steps were taken towards reaching our most ambitious destination yet.
It’s incredible to finally say that you, the reader, can now use System Initiative in its generally available, fully-featured form.
Replacing Infrastructure as Code Link to heading
We’ve talked about System Initiative many times before, but this time is different: we’re now talking about System Initiative in the form that you deserve; over five years in the making.
System Initiative is 100% Open Source, fully multiplayer, and replaces infrastructure as code. It provides a living architecture diagram to manage all of your infrastructure, no matter where it lives. It offers full programmability and customization, and you can get started with over 100 rich models that are 1:1 with upstream APIs, like AWS EC2.
I wrote about the capabilities of the System Initiative software in the past, but a lot has improved and changed since then. Let’s talk about it in two parts: features and performance.
Feature Forward Initiative Link to heading
We provide over 100 assets that richly model upstream APIs 1:1, but what if you want to write your own? What if you want to contribute your own? Maybe you have a small edit to make to an existing asset, just for you. Maybe you want to share your assets with your colleagues. What if you want to change the inputs for functions that your assets use? Maybe you want to add an additional socket to your asset? You can do all of this and more in System Initiative.
Configuration and modeling were first class features in early versions of System Initiative. However, we know that System Initiative thrives when it is customized with maximum programmability. Over the past year, authoring has not only become a first class feature of SI, but it has become a cornerstone of what makes SI transformative and truly special.
Many low-code tools suffer once the honeymoon is over and you need to customize your experience to your needs. System Initiative embraces the power user and provides more flexibility than infrastructure as code has ever done before. Authoring assets and functions allows you to interact with upstream APIs directly without having to deal with abstractions.
Need to model a virtual machine from a cloud provider? You can model their API 1:1 by using the asset builder and authoring interface. Need to provide metadata from the virtual machine to another component on the diagram? You can add an output socket to your new asset and source the value from another prop within the asset. Need to transform the data on the way out into a different shape? You can write a custom attribute function to transform the data for other components to use.
Don’t worry, if this terminology is new or confusing, we have tutorials and docs ready for you today at docs.systeminit.com! You can also reach out to us and fellow community members anytime in our Discord server.
Configuration works better and faster than ever too. You can compose secrets throughout the system and have secrets that depend on secrets to be used in other components. You can create complex, nested frames to mass connect sockets without having to draw individual edges. You can see and learn more from the diagram without having to navigate between different screens and views, more than ever before.
Whether you make small edits to a couple properties on a single component or create a hundred AWS EC2 key pairs at once (totally not a performance smoke test of mine), you will be able to compose, configure and manage your infrastructure on one screen to rule them all. The best part? You can do this with your friends and colleagues in real-time.
Say goodbye to pull request ping pong and waiting for long CI/CD queues! Your configurations can be validated and qualified automatically through customizable functions. Not only is deployment native and abstraction-less, but the collaboration we’ve all been craving is now there.
This is all powered by a living architecture diagram that reflects both the real world through imperative changes and your model through declarative changes.
Infrastructure is a living, breathing thing. It changes, drifts, crashes, reboots, upgrades, and shifts. Declarative infrastructure is great, but giving you the choice is essential. We deserve to have the choice.
Performance Enhanced System Link to heading
Since the open source launch, performance has dramatically improved through a variety of means; one of which is through our homegrown naxum library. It’s an axum-like server that handles core NATS and NATS JetStream messages at lightning fast speeds.
All of our services communicate via core NATS for fan-out and UDP-style messaging, and NATS JetStream for workqueue and durability-sensitive messaging. Creating naxum was a labor of love by Fletcher Nichol and many others at SI, and it’s been incredible for building an intensely complex distributed system.
From late July 2023 to March 2024, we worked on a new engine for System Initiative. Powered by petgraph, the new engine switches from a hypergraph of functions powered by PostgreSQL tables and complex queries to an in-memory, directed acyclic graph backed by a multi-layered, distributed database. Complex operations moved out of PostgreSQL moved into our own services, providing substantial speed increases. Questions about lineage for updating existing assets with new versions upstream and providing a fully-featured multiplayer experience became infinitely easier to answer.
Originally, we embarked on the new engine journey to tackle the lifecycle of modules. There was no talk of a “new engine” at the time. We needed to handle the lineage of modules and assets in workspaces and change sets. However, it quickly grew into a journey about performance, features and ultimately, a paradigm shift at the engine-layer of the System Initiative software.
Since May 15th of this year, we’ve been fine tuning and hardening the new engine since reaching feature parity with the old engine. In fact, we’ve since made our graphs simpler, dropped their reliance on vector clocks, and have switched to an operational-transform model for eliminating needless conflicts across the board.
The storage layer is powered by our handcrafted layer caching service. The layers include both memory caching and disk caching, and all writes are eventually written to PostgreSQL.
All of these performance optimizations are for naught without horizontal scaling capabilities and effective multi-core utilization. All SI backend services can be horizontally scaled with 1:N instances. They feature both graceful shutdown procedures and are protected against unexpected shutdown scenarios. This is accomplished through a variety of means, but using NATS JetStream for durable messaging, and our layer caching service for fast reads and writes, helps ensure that our distributed system is both fast and correct.
Go try it! Link to heading
Over five years of handcrafting the future of DevOps automation has been an incredibly tall task, but we are more than excited to give you what our industry needs now, more than ever.
If you made it this far, you owe it to yourself to go to our website and try it. No need for me to keep yappin’ here.
Thank you, once again, to my team and our community for everything you’ve done. As always, I’ll see you on our Discord server!