Before we get started taking a look at the tech stack that we’ve built for satsearch, let me introduce myself. My name is Alberto Vaccarella, and I’m the tech lead at satsearch. I’ve got a PhD in Bioengineering from Politecnico di Milano and I’ve worked at various companies as a software developer.
When Kartik approached me in 2015 about building a “Google for space”, I was stunned to learn that a literal frontier industry has not yet solved this problem. Kartik shared his frustration as an Aerospace engineer, spending hours Googling for information about suppliers and products on the market.
And so satsearch was born!
This article is the first in a series that will touch on the technologies we’re employing at satsearch to build the first global marketplace for space. Luckily, tech companies have solved many of the challenges we face over the last couple of decades, so we’re leveraging a vast ecosystem of tools and processes to build and grow satsearch.
The origin story for satsearch starts in April 2015. Kartik and I built and launched v1 of the website in under 48 hours. The beauty of building websites today is that there is a rich ecosystem of tools that you can use to get started right away.
v1 was built using the MEAN stack, minus Angular as a frontend framework. This allowed us to get started quickly. The choice for Node.js in particular was based on our thought that having the same language for both the server and client sides of our website would be a huge benefit. We did briefly consider other options like Ruby on Rails and Python via Django or Flask, but the immediate benefit of working in a single language won out in the end. Using Express meant that we could get started quickly, with a lightweight web framework.
We decided not to adopt Angular or any other frontend framework, to reduce the learning curve. Instead, we made use of the Jade template engine (since renamed Pug) for Node and Bootstrap. This allowed us to build a lightweight frontend that included some dynamic content, without having to adopt a full framework. Bootstrap, an open source toolkit for developing with HTML, CSS, and JS, is especially useful when prototyping ideas, and has come a long way since it was initially released by Twitter in 2010.
We deployed the site to Heroku the first weekend it was built. Heroku is a great service to prototype new ideas, especially since their free tier is pretty generous. The Git deploy workflow on Heroku is also great if you’re frequent/heavy Git users like us.
For v1, we thought that the best way to be the “Google for space” was to look like the “Google for space”. Over the course of 2015 and 2016 however, we discovered that this limited the user experience. This led us to the reboot, at the start of 2017.
After a lot of user feedback, we decided that the website needed a reboot. Instead of trying to mimic Google (which turns out to be very difficult), we decided to build something that was more akin to Digi-Key, since this seemed to address what our users wanted. So we set about rethinking the way our website functioned and mapping out the user journey.
At the start of 2017, we launched our reboot: v2 of satsearch.
This, in broad strokes, is the user interface (UI) that we are working with today. After the reboot, we focussed the rest of 2017 on behind-the-scenes work. In particular, we had to build a robust data pipeline to support a growing number of users and searches on the website.
We discovered that search was becoming more and more of an issue as the website grew, especially with our database growing from a few hundred products to thousands by the end of 2017. So the question that we had to address was how to build search that would enable users to find what they were looking for, as quickly as possible.
Data modeling & search
One of the fundamental challenges that we face is finding a way to build a system that consistently and reliably provides users with relevant search results. The problem of building good search is effectively a solved problem (you can even follow online courses that take you through the whole journey). Nevertheless, the devil is in the details. In our case, we had to think about how to model the underlying data to provide users with a powerful and intuitive way to explore the global space supply chain.
We decided early on to capture information about suppliers, products, and services as JSON documents. JSON was originally developed in the late 90s as a lightweight data-exchange format for the web that’s easy for both humans and machines to read. Since the early days, JSON has been adopted across the web and is the standard language of Application Programming Interfaces (APIs) that power pretty much every website you interact with on a daily basis.
Given that we decided to think of our underlying data in terms of documents, it seemed like a natural fit to start prototyping with MongoDB (the “M” in the MEAN stack). MongoDB is one of a large number of NoSQL database technologies that represents data as JSON-like documents. The power of using a document-based database is that it allows fields to vary from document to document, meaning you don’t have to fix a rigid schema a priori. This seemed like the right choice, as we had to discover structure in the data that we were collecting about supplier and products, meaning that we didn’t know the schema ahead of time.
As we continued to grow our website though, we discovered that the fundamental issue that we were facing was that our data was highly relational in nature, meaning that we were actually reinventing the wheel in our application, by imposing rules that come for free in a relational database. This caused inconsistencies and resulted in huge bloat in our codebase, which became unmanageable. Hence, we took the big decision to migrate our backend to PostgreSQL, a popular, open-source relational database. This migration is on-going and will be finalized by the end of the summer, enabling us to scale our backend much more easily.
Lesson learned: don’t get sucked into thinking that fancy, shiny new technologies are the best answer; sometimes the stable, robust oldies are still the right choice.
There are many challenges that lie ahead, as we continue to grow. Some of the challenges we’re addressing include:
- Improved search
- A consistent and scalable frontend
- Scalable and (semi-)automated data modeling
In the coming months, I’ll be shedding more light on these challenges and share an in-depth perspective on key choices for our tech stack and product.
If you’re interested in learning more about the technologies we’re using at satsearch, or if you have suggestions/feedback, I’d love to hear from you. If you’re also thinking of prototyping your own idea and we can help with some do’s and don’ts, don’t hesitate to reach out.
Feel free to join our Slack community, or email me at [email protected].