What’s Next For Subsquid? A Look Back and a Look Forward

TL;DR: After the listing, we are up to a lot. We’ll launch the mainnet and add support for further HighTPS chains, Cosmos, and DAs. We’re also working on further decentralizing data submission and streaming from the data lake to reach our goal of scaling data access for all of Web3. 

They say all roads lead to Rome. For us, Rome is becoming the data access layer for Web3. Yet, an important step on the road there is to make our token widely available and easily accessible for devs, and community members. 

With the listing of our SQD token on 5 exchanges we’re also putting out part of the results of our labor for the past 3 years. People often walk up to us saying they’d never heard about Subsquid. They react surprised when they learn that we’ve been around for 3 years (an eternity in crypto).

You know what they also say about Rome? It wasn’t built in a day. 

We’ve been continuously working on enhancing the functionality of our product and onboarding builders across ecosystems to grow Subsquid’s market share while empowering developers regardless of where they build. 

A big event, such as a listing, is a great moment to take a walk down memory lane. You might be surprised by what you find. 

Subsquid’s early days 

In its earliest days, Subsquid wasn't even called by its current name. It all began as part of Joystream, a project building a decentralized video streaming platform on Substrate. One of the biggest challenges to delivering videos at scale is data. To enable quick file delivery and not leave viewers hanging, they built a next-gen query framework called Hydra. 

It quickly became apparent that a query framework isn't just useful for video streaming but for countless web3 use cases. Fast forward, and Hydra spun out of Joystream to become what we now know as Subsquid. 

The initial goal of building the best tool for indexing on-chain data for teams building on Substrate was accomplished and replaced with the larger mission to enable any web3 builders to run their own Squids (indexers). 

With 17.5 million in total funding, we grew further and expanded to serve builders on more than 150 networks with reliable, fast access to on-chain data. Last year, we launched our first testnet to validate our thesis and build a secure, fast, scalable data access layer. 

Our testnet

Mirovia Era 

The first phase of our testnet started in September 2023 and concluded at the beginning of 2024. Throughout this phase, our community enabled us to test resilience under load. Over the three months, 50k indexers were deployed, sourcing data from Ethereum, Polygon, Arbitrum, and more. Thanks to everyone joining and pushing us to 450,000 completed quests - many of which put further strain on the network. 

The Panthalassa Era

During this phase, we were testing the security of the mechanisms required for network rewards, including sending rewards to worker nodes and the delegation mechanism. 

Our testnet ended on April 1st and was a huge success. It allowed us to battle-test the functionality of Subsquid’s infrastructure and set us up for the upcoming mainnet launch. 

What else have we been up to 

Of course, we didn’t just sit around during the test phase. Instead our devs were working tirelessly in their basements to integrate new chains, and the rest of our team pushed for meaningful integrations and partnerships. 

Highlights include: 

• Becoming the first decentralized indexing solution for Solana with the Launch of our Solana Beta.
• Adding support for countless Bitcoin Layer-2s 
• Collaborating with Google Cloud with an integration allowing devs to analyze data from our Data Lake using Google’s tools 
• Expanding our partner network to over 170 and integrating a total of 150 + chains, L2s, and DA Layers. 

Check out the graphic below for a holistic overview of partners. Our partners cover all of crypto, from infra to DePIN and, more recently, as a hype area, AI. 

What’s next? 

Mainnet launch 

As you probably figured with the listing, the mainnet launch is imminent. Soon, we’re releasing the Squids into the wild world of mainnet. 

This lays the foundation for future growth of the network, allowing users to delegate tokens and devs to run and access data permissionlessly. While this might mark another big milestone, it's far from the final step in our journey. After the mainnet launch, we'll add further features such as: 

Cosmos support 

Cosmos has been modular with its thesis before it was cool. The ecosystem comprises various blockchains that all use the IBC SDK to spin up their own custom chains but communicate through the Cosmos Hub. This network is also referred to as the internet of blockchains and has been home to some great DeFi and other crypto experiments. By adding shared security, it's become even more accessible to run one's own chain, and since Cosmos is community-operated, it's also been an insightful place to watch for anything governance-related. 

While the drama might sometimes deflect, the tech is great and used outside of Cosoms, to name just one big example, for DA with Celestia. Plus, they are winning when it comes to visualizing their ecosystem.

By adding Cosmos support, builders using Subsquid can tap into all that's happening on-chain in the internet of blockchains for multi- and cross-chain dApps that unite rather than divide. 

Other VMs

And we’re not just planning to add Cosmos. In crypto, there is always a new Virtual Machine just around the corner that promises to alleviate some of the pain points of the EVM. With experience in integrating with new VMs, we’ve got a few in the pipeline. Among other things, we’ll add the OG chain Bitcoin to our data lake. Another newer VM we'll be supporting is Move, the language that has been making waves recently.

And then some other upcoming, innovative VMs. Stay tuned for those. 👀

Permissionless dataset submission 

Currently, Subsquid GmbH, the company behind Subsquid, maintains, curates, and validates the datasets that are submitted to the data lake. However, we're well aware that a true web3 data lake cannot just rely on a single party's contributions. Consequently, we'll be opening access to submitting datasets to anyone; no permission is required. This will enhance data diversity by allowing a wider array of sources and benefit consumers with a more comprehensive pool of information.

Decentralized SQL database streaming 

In general, database streaming refers to distributing and syncing databases across nodes of the datalake. However, unlike in traditional SQL setups where databases are stored on a single server, with Subsquid information is distributed. Ensuring it's accurate and up to date at all times requires a continuous flow of changes captured as a stream of events. 

This is what database streaming enables. It facilitates continuous sync and propagation of data across the entire data lake in a fault-tolerant and scalable manner. 

Enterprise tooling 

The last step on our current roadmap is the implementation of two enterprise data tools: 

• Kafka is an open-source, distributed event streaming platform that allows enterprises to handle large amounts of data in real-time. It's used by countless tier-1 companies such as Netflix, Uber, and Paypal, enabling them to run real-time analytics and monitoring, among other things. 
• Snowflake is a cloud-based data platform offering tools for data integration, transformation, and analysis of big data. With Snowflake, companies can store and analyze data at scale, eliminating manual management tasks. 

By integrating these tools, Subsquid data will become accessible to enterprises beyond web3 that want to tap into blockchain data. At the same time, crypto projects can use streaming platforms like Kafka for their own real-time analytics and monitoring use cases. 

Thanks for supporting Subsquid! 🦑