Sign in Subscribe

Get Started Guide: Gateway Runners (Data Consumers)

SQD Editor

• 5 min read

In a rush? Click here to go straight to the how-to guide.

Subsquid is a data-access layer for web3, aiming to make historical data available for dapps and individual developers. It works as a decentralized network of nodes, each storing a chunk of data. However, to make this data useful, it is important to make it available for all. This is where the gateway runner comes in.

When a user sends a query to the network, it goes through a gateway and is sent to the network. After the relevant node is found, it sends the response with the data to the user. A gateway runner is a connecting link between the decentralized data lake and anyone who needs the data. 

Gateway Runners

Note at the top → If you would just like to get data from Subsquid Network without running a gateway, consider using Subsquid Cloud. 

Gateway runners (Data consumers) use the network to gain access to verified historical blockchain data. They operate a network gateway or use an external service to access the network. Data consumers are usually dapps that rely on historical data.

A gateway provides a connection to the Subsuid Network. Gateways are bound to on-chain addresses. The number of requests that each gateway can submit to the network is capped by a number calculated based on the number of locked $SQD tokens. Thus, the more tokens the gateway operator locks, the more bandwidth it is allowed to consume.

To fetch data, you can rely on a public or private gateway or run your own. Running your gateway gives you control over how you want to operate and access the network. It also reduces costs by eliminating third-party providers.

How to Run a Gateway

Running a gateway enables you to access Subsquid Network data without relying on any centralized services. You can run a private gateway for your own needs, or a high-throughput public gateway.

To get started, you will need

  • a working Docker installation
  • some SQD tokens (in your wallet or in a special vesting contract)
  • some Arbitrum ETH (for gas)

All gateways have to be registered on-chain and have a SQD stake associated with them to begin serving requests. Size and duration of the stake determine the rate at which the gateway can serve requests.

To start a gateway, Generate your key file by running

docker run --rm subsquid/rpc-node:0.2.5 keygen > <KEY_PATH>

The command will display your peer ID:

Your peer ID: <THIS IS WHAT YOU NEED TO COPY>

Please copy this ID, as it will be needed for further steps.

  1. Go to https://network.subsquid.io.
  2. Connect your EVM wallet (we recommend using Metamask). Use the wallet holding the tokens or linked to the vesting contract.
  3. Go to the "Gateways" tab and click the "Add gateway" button.

Fill out the gateway registration form.

Gateway registration form

If you plan to make your gateway public, click the "Publicly available" switch and populate the additional fields.

Gateway registration form - public
  1. Once done, click "Register" and confirm the transaction.
  2. Go to the "Gateways" tab and click the "Get CU" button. Stake the amount of SQD appropriate for the planned bandwidth of your gateway (see Staking requirements and compute units).
  3. Wait for your stake to become active. This will happen at the beginning of the next epoch.
  4. Clone the gateway repo and enter the folder.

git clone git clone https://github.com/subsquid/query-gateway

  1. cd query-gateway
  2. Prepare the environment. Begin with
  3. cp mainnet.env .env # use tethys.env for the testnet
  4. then set the path to your key file:
  5. echo KEY_PATH=<KEY_PATH> >> .env
  6. Run your gateway. You can either utilize a pre-built Docker image:
  7. docker compose up -d
  8. or build it from source:
  9. cargo run --release

What does it cost?

The cost is calculated in “Compute Units”. Compute units is a form of credits used to submit queries to the Subsquid Network. Locked SQD tokens yield virtual "compute units" (CU) based on the period the tokens are locked. All queries cost the same price of 1 CU (until complex SQL queries are implemented). 

To incentivize gateway operators, The locking mechanism has an additional booster design to encourage them to lock the tokens for longer periods of time. The longer the lock period, the more CUs are allocated per SQD/yr. For example, if a gateway operator locks 100 SQD for 2 months, the virtual yield is 2SQD, which means it can perform 8000 queries (8000 CU). If 100 SQD are locked for 3 years, the virtual yield is 12% * 3 APY, so the operator gets CUs worth 108 SQD; that means it can submit up to 432000 queries to the network within the period of 3 years.

How to Access Data in Subsquid Network

First option is to use the Squid SDK .Squid SDK allows developers to create an indexing project and to retrieve and process blockchain data from the Subsquid Network. The Squid SDK is a set of open-source Typescript libraries that retrieve, decode, transform, and persist the data. It can also make the transformed data available via an API. All stages of the indexing pipeline, from the data extraction to transformation to persistence, are performed on batches of blocks to maximize the indexing speed. The modular architecture of the SDK makes it possible to extend indexing projects (squids) with custom plugins and data targets.

Subsquid Network also provides an API to access blockchain data directly. It is possible to use this API to fetch data through Terminal with `curl`. See this guide for step-by-step instructions.

Subsquid Network API can also be used to create your own developer tooling to fetch historical blockchain data.

With Subsquid Network it is possible to run a subgraph without a full node. Subsquid’s Firehose adapter runs as a side-car to a graph indexer node, ingesting and filtering the data directly from Subsquid Network instead of an RPC endpoint.To migrate your subgraph,check out this doc. 

Another option is ApeWorx. It is a modular Web3 development framework for Python programmers. Among other things, it is capable of retrieving blockchain data in bulk. The data can come from various sources, including Subsquid Network. To use the Subsquid Network with ApeWorx, follow this tutorial.

Finally, you can use DipDup, a Python framework for building smart contract indexers. It operates on top of the Subsquid Network, using it as a source for historical data. 

For a more in-depth explanation of what we’re up to at Subsquid, check out the Whitepaper.