Why Rust?

tl;dr Rust’s ability to run anywhere makes this low-level language adept at solving high-level problems: it's fast, easy to integrate, and serves as an excellent foundation for any data ecosystem.

Rust is the new, flashy language in town, and while as engineers we're not entirely immune to hype, we believe Rust is the right platform for us, and here's why. At its core, Quary is an open-source SQL templating engine. It lets you write, test, and organize SQL queries, letting you compose queries to build complex transformations. These run in your data warehouse and helps you derive insights.

Templating is not the slow part of the flow, so why Rust? We believe that its speed in other dimensions is what truly matters; dimensions more critical for our users and the future ecosystem.

For our users

When people question the use of Rust for our templating engine, we often feel like their mental model is the following:

It’s simple, suppose you run a data transformation, templating the SQL takes an inconsequential amount of time compared to running a query. Optimising that is a waste of time. While that perspective is accurate, we argue it’s the wrong picture to concentrate on. 'Data transformation' constitutes only a fraction of the data workflow:

There are many other steps to getting a data insight that are equally, if not more, time-consuming. Here are a few examples: deciding the initial requirements, finding the right tool, security reviews, setting up environments, reviewing the code that defines the data transformation layer, data quality spot checks, updating data catalogs, waiting for merges to build reports, internal marketing for that data asset... the list goes on. It’s some of these high-level components we believe Rust tackles best. Let’s take a few examples:

Trying tools 🛠️: We like to try tools before committing to one of them. With Quary, you can do so with less risk, and you don't have to update your customers about new sub-processors. Our core engine runs on your browser, making sure your data doesn't go through Quary servers.

Setting up environments 💻: For those who don't focus daily on data, but occasionally dip in, Quary runs on your browser and allows you to access its capabilities without setting up a new, local environment.

Reviewing code and data quality checks 🔍: When reviewing code, it’s often useful to open the change, play with it, and perform a few spot checks. With Quary, that process is instant: open a PR in our catalogue or visual studio code and run the necessary checks right on your browser.

Analogy with Figma

We see significant parallels with Figma. Remember the days before Figma, when a Product Manager (PM) would get involved to make edits and update copy to avoid painful back and forth? We don’t either because of those pesky barriers to entry. Figma lowered the bar to entry and changed how design in organisations works.

We sometimes think of as Quary like Figma for analytics engineering.

For us

We recently started to build a data catalog/explorer in the browser. Our web app parses the GitHub repository and allows you to search and run queries. It's how we hope data engineers can share their work with their team. We built it in Rust and WebAssembly. The simplicity in not needing other servers/services has been a massive win for us and thus our users.

For our ecosystem

The data ecosystem thrives because data tools work well together. One of the biggest reasons we decided to build our core in Rust was to make it easier for our future partners to integrate with Quary. Just like our core engine made it easier for us to build our data catalog, we hope it will make it easier for others to build on top of Quary. We really enjoy building our data catalog with Wasm, and hope you do too!

It's early, but we're excited about the possibilities of creating bindings for other languages, running Quary in the browser, and building a community around it. We think Rust is a good building block for that.

Downsides

I’d be remiss if I didn’t talk about the downsides of Rust. Rather than go into detail, I thought I’d link to this article, which I think explains the downsides better than I could.

Slower iterations. When you first start with Rust, you’ll end up fighting the compiler a bit. This is natural and gets easier over time. However, at a new startup, one of the core problems is to figure out if you are building something useful.

Rust isn’t all sunshine ☀️ and roses 🌹, but it's a compromise which is well worth it in the end!