From 8f7b125ce69704029cb45f1b679014ce3d3267ff Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:01:20 -0400 Subject: [PATCH 01/30] Create ArchitecturalOverview.md --- pages/Architecture/ArchitecturalOverview.md | 46 +++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 pages/Architecture/ArchitecturalOverview.md diff --git a/pages/Architecture/ArchitecturalOverview.md b/pages/Architecture/ArchitecturalOverview.md new file mode 100644 index 00000000..bf9b824f --- /dev/null +++ b/pages/Architecture/ArchitecturalOverview.md @@ -0,0 +1,46 @@ +# Intro to dYdX v4 Architecture + +v4 (or dYdX Chain) is the next iteration of the dYdX protocol, which will consist of open-source software. The current version live in production today is known as v3 and can be found [here](https://trade.dydx.exchange/?utm_source=dydx-website&utm_medium=blog&utm_content=v4-technical-architecture-overview). v3 and past versions of dYdX were, at their core, smart contracts deployed to existing chains, combined with centralized services hosted in the cloud. v4 will be a standalone L1 blockchain, featuring a fully decentralized, off-chain orderbook and matching engine. The dYdX Chain will be based on the Cosmos SDK and CometBFT PoS consensus protocol. + +As we approach the release of v4 mainnet open-source software (“dYdX Chain”), we want to provide a peek into what the dYdX team is building. This post presents a high-level overview of the architecture of v4. Given that v4 is still under development, please note that what is presented below is subject to change. + +### v4 System Architecture +dYdX v4 is being designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components will be made available as open source software. None of the components will be run by dYdX Trading Inc. + +![image](https://github.com/dydxprotocol/v4-documentation/assets/130097657/e9a54253-e7fa-44ab-97c5-ae1ce7cae320) + +### Protocol (or “Application”) +The open-source protocol is an L1 blockchain built on top of [CometBFT](https://dydx.exchange/blog/v4-technical-architecture-overview#:~:text=on%20top%20of-,CometBFT,-and%20using%20CosmosSDK) and using [CosmosSDK](https://v1.cosmos.network/sdk). The node software is written in Go, and compiles to a single binary. Like all CosmosSDK blockchains, v4 uses a proof-of-stake consensus mechanism. + +The protocol will be supported by a network of nodes. There are two types of nodes: + +- **Validators**: Validators are responsible for storing orders in an in-memory orderbook (i.e. off chain and not committed to consensus), gossipping transactions to other validators, and producing new blocks for the dYdX Chain through the consensus process. The consensus process will have validators take turns as the proposer of new blocks in a weighted-round-robin fashion (weighted by the number of tokens staked to their node). The proposer is responsible for proposing the contents of the next block. When an order gets matched, the proposer adds it to their proposed block and initiates a consensus round. If ⅔ or more of the validators (by stake weight) approve of a block, then the block is considered committed and added to the blockchain. Users will submit transactions directly to validators. + +- **Full Nodes**: A Full Node represents a process running the v4 open-source application that does not participate in consensus. It is a node with 0 stake weight and it does not submit proposals or vote on them. However, full nodes are connected to the network of validators, participate in the gossiping of transactions, and also process each new committed block. Full nodes have a complete view of the dYdX Chain and its history, and are intended to support the Indexer. Some parties may decide (either for performance or cost reasons) to run their own full node and/or Indexer. + +### Indexer +The Indexer is a read-only collection of services whose purpose is to index and serve blockchain data to end users in a more efficient and web2-friendly way. This is done by consuming real time data from a v4 full node, storing it in a database, and serving that data through a websocket and REST requests to end-users. + +While the v4 open-source protocol itself is capable of exposing endpoints to service queries about some basic on-chain data, those queries tend to be slow as validators and full nodes are not optimized to efficiently handle them. Additionally, an excess of queries to a validator can impair its ability to participate in consensus. For this reason, many Cosmos validators tend to disable these APIs in production. This is why it is important to build and maintain Indexer and full-node software separate from validator software. + +Indexers will use Postgres databases to store on-chain data, Redis for off-chain data, and Kafka for consuming and streaming on/off-chain data to the various Indexer services. + +### Front-ends +In service of building an end-to-end decentralized experience, dYdX is building three open-source front ends: a web app, an iOS app, and an Android app. + +- **Web application**: The website will be built using Javascript and React. The website will interact with the Indexer through an API to get off-chain orderbook information and will send trades directly to the chain. dYdX will open source the front end codebase and associated deployment scripts. This will allow for anyone to easily deploy and access the dYdX front end to/from their own domain/hosting solution via IPFS/Cloudflare gateway. + +- **Mobile**: The iOS and Android apps are built in native Swift and Kotlin, respectively. The mobile apps will interact with the Indexer in the same way the web application does, and will send trades directly to the chain. The mobile apps will be open sourced as well, allowing anyone to deploy the mobile app to the App Store or Play store. Specifically for the App store, the deployer will need to have a developer account as well as a Bitrise account to go through the app submission process. + +### Lifecycle of an Order +Now that we have a better understanding of each of the components of dYdX v4, let’s take a look at how it all comes together when placing an order. When an order is placed on v4, it will follow the flow below: + +1. User places a trade on a decentralized front-end (e.g., website) or via API +2. The order is routed to a validator. That validator gossips that transaction to other validators and full nodes to update their orderbooks with the new order. +3. The consensus process picks one validator to be the proposer. The selected validator matches the order and adds it to its next proposed block. +4. The proposed block continues through the consensus process. + 1. If ⅔ of validator nodes vote to confirm the block, then the block is committed and saved to the on-chain databases of all validators and full nodes. + 2. If the proposed block does not successfully hit the ⅔ threshold, then the block is rejected. +5. After the block is committed, the updated on-chain (and off-chain) data is streamed from full nodes to Indexers. The Indexer then makes this data available via API and Websockets back to the front end and/or any other outside services querying for this data. + +The flow above is a high level overview of how an order/data will move through v4. We will do further deep dives into the protocol, indexer, and various front ends’ infrastructure in subsequent blog posts as we get closer to the release of v4 mainnet open-source software. From 2ab1d0106c93b5d7c2cb1e08b86f58c933deb99c Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:03:30 -0400 Subject: [PATCH 02/30] Create Indexer.md --- pages/Architecture/Indexer.md | 102 ++++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 pages/Architecture/Indexer.md diff --git a/pages/Architecture/Indexer.md b/pages/Architecture/Indexer.md new file mode 100644 index 00000000..4b8f2a3f --- /dev/null +++ b/pages/Architecture/Indexer.md @@ -0,0 +1,102 @@ +# Indexer Deep Dive + +A good way to think about the indexer is as similar to Infura or Alchemy’s role in the Ethereum ecosystem. However, unlike Infura/Alchemy, and like everything else in dYdX v4, the Indexer is completely open source and can be run by anyone! + +### What is the Indexer? + +As part of tooling for the dYdX ecosystem, we want to ensure that clients have access to performant data queries when using the dYdX v4 exchange. Cosmos SDK Full Nodes offer a number of APIs that can be used to request on-chain data. However, these Full Nodes are optimized for committing and executing blocks, not for serving high frequency, low-latency requests from web/mobile clients. + +This is why we wrote software for an indexing service. The Indexer is a read-only service that serves off chain data to clients over REST APIs and Websockets. Its purpose is to store and serve data that exists on the dYdX Chain in an easier to use way. In other words, the purpose of an indexer is to index and serve data to clients in a more performant, efficient and web2-friendly way. For example the indexer will serve websockets that provide updates on the state of the orderbook and fills. These clients will include front-end applications (mobile and web), market makers, institutions, and any other parties looking to query dYdX Chain data via a traditional web2 API. + +### On-chain vs. Off-chain data + +The Indexer will run two separate ingestion/storage processes with data from a v4 Full Node: one for on-chain data and one for off-chain data. Currently, throughput of on-chain data state changes is expected to be from 10-50 events/second. On the other hand, the expected throughput of off-chain data state changes is between 500-1,000 events/second. This represents a 10-100x difference in throughput requirements. By handling these data types separately, v4 is built to allow for different services to better scale according to throughput requirements. + +### On-chain Data + +On-chain data is all data that can be reproduced by reading committed transactions on the dYdX Chain. All on-chain data has been validated through consensus. This data includes: + +1. Account balances (USDC) +2. Account positions (open interest) +3. Order Fills + 1. Trades + 2. Liquidations + 3. Deleveraging + 4. Partially and completely filled orders +4. Funding rate payments +5. Trade fees +6. Historical oracle prices (spot prices used to compute funding and process liquidations) +7. Long-term order placement and cancellation +8. Conditional order placement and cancellation + +### Off-chain Data + +Off-chain data is data that is kept in-memory on each v4 node. It is not written to the blockchain or stored in the application state. This data cannot be queried via the gRPC API on v4 nodes, nor can it be derived from data stored in blocks. It is effectively ephemeral data on the v4 node that gets lost on restarts/purging of data from in-memory data stores. This includes: + +1. Short-term order placement and cancellations +2. Order book of each perpetual exchange pair +3. Indexed order updates before they hit the chain + +## Indexer Architecture + +![image](https://github.com/dydxprotocol/v4-documentation/assets/130097657/8fc9842f-49e7-430f-a1f0-969c72489b28) + +The Indexer is made up of a series of services that ingest information from v4 Full Nodes and serve that information to various clients. Kafka topics are used to pass events/data around to the services within the Indexer. The key services that make up Indexer are outlined below. + +### Ender (On-chain ingestion) + +Ender is the Indexer’s on-chain data ingestion service. It consumes data from the “to-ender” Kafka topic (which queues all on-chain events by block) and each payload will include all event data for an entire block. Ender takes all state changes from that block and applies them to a Postgres database for the Indexer storing all on-chain data. Ender will also create and send websocket events via a “to-websocket-?” Kafka topic for any websocket events that need to be emitted. + +### Vulcan (Off-chain ingestion) + +Vulcan is the Indexer’s off-chain data ingestion service. It will consume data from the “to-vulcan” Kafka topic (queues all off-chain events), which will carry payloads that include active order book updates, place order updates, cancel order updates, and optimistic fills. This data will be stored in a Redis cache. Vulcan will update Redis with any new open orders, set the status of canceled orders to cancel pending, and update orderbooks based on the payload received. Vulcan will also update Postgres whenever a partially filled order is canceled to update the state of the order in Postgres. Vulcan will also create and send websocket events via a “to-websocket-?” Kafka topic for any websocket events that need to be emitted. + +### Comlink (API Server) + +Comlink is an API server that will expose REST API endpoints to read both on-chain and off-chain data. For example, a user could request their USDC balance or the size of a particular position through Comlink, and would receive a formatted JSON response. + +As an explicit goal set out by the dYdX team, we’re designing v4 APIs to closely match the [v3 exchange APIs](https://dydx.exchange/blog/v4-deep-dive-indexer#:~:text=closely%20match%20the-,v3%20exchange%20APIs,-.%20We%20have%20had). We have had time to gather feedback and iterate on these APIs over time with v3, and have confidence that they are reasonable at the product-level. + +### Roundtable + +Roundtable is a periodic job service that provides required exchange aggregation computations. Examples of these computations include: 24h volume per market, open interest, PnL by account, candles, etc. + +### Socks (Websocket service) + +Socks is the Indexer’s websockets service that allows for real-time communication between clients and the Indexer. It will consume data from ender, vulcan, and roundtable, and send websocket messages to connected clients. + +## Hosting & Deploying the Indexer + +In service of creating an end-to-end decentralized product, the Indexer will be open source. This will include comprehensive documentation about all services and systems, as well as infrastructure-as-code for running the Indexer on popular cloud providers. + +The specific responsibilities of a third party operator looking to host the Indexer generally include initial deployment and ongoing maintenance. + +Initial deployment will involve: + +- Setting up AWS infrastructure to utilize the open-source repo. +- Deploying Indexer code to ingest data from a full-node and expost that information through APIs and websockets +- Datadog (provides useful metrics and monitoring for Indexer services), and Bugsnag (real-time alerting on bugs or issues requiring human intervention). + +Maintenance of the Indexer will involve: + +- Migrating and/or upgrading the Indexer for new open-source releases +- Monitoring Bugsnag and Datadog for any issues and alerting internal team to address +- Debugging and fixing any issues with a run book provided by dYdX + +dYdX believes that, at minimum, a DevOps engineer will be required to perform the necessary duties for deployment and maintenance of the Indexer. An operator will need to utilize the services below: + +- AWS + - ECS - Fargate + - RDS - Postgres Database + - EC2 + - Lambda + - ElastiCache Redis + - EC2 ELB - Loadbalancer + - Cloudwatch - Logs + - Secret Manager +- Terraform Cloud - for deploying to the cloud +- Bugsnag - bug awareness +- Datadog - metrics and monitoring +- Pagerduty - alerting + +Operators should be able to host the open-sourced Indexer for public access in a highly available (i.e., high uptime) manner. Requirements include owning accounts to the services above and hiring the appropriate personnel to perform deployment and maintenance responsibilities. From cf12c007a85f08b2e5e4c6e75a182c68c965fd0c Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:04:32 -0400 Subject: [PATCH 03/30] Update _meta.json --- pages/_meta.json | 1 + 1 file changed, 1 insertion(+) diff --git a/pages/_meta.json b/pages/_meta.json index 5989267a..4b634bb3 100644 --- a/pages/_meta.json +++ b/pages/_meta.json @@ -1,6 +1,7 @@ { "index": "Introduction", "about": "About dYdX", + "architecture": "Architecture" "validators": "Validators", "network": "Network Configuration", "trading": "Trading", From fe517b4aa8a25c19e577c09d63ebca8c6319c71c Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:08:52 -0400 Subject: [PATCH 04/30] Create FAQ.md --- pages/FAQ.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 pages/FAQ.md diff --git a/pages/FAQ.md b/pages/FAQ.md new file mode 100644 index 00000000..8c40adb2 --- /dev/null +++ b/pages/FAQ.md @@ -0,0 +1,7 @@ +# FAQ Documentation + +Since the inception of v4, we have been compiling a list of relevant questions and answers that are asked by MMs/Trading Firms on Telegram. We turned this list into a live _v4 Telegram Questions_ document, which should be able to answer many of your more nuanced questions. + +It's a large document, so we recommend CTRL+F to find key words. + +You can anonymously access the live document with [this link](https://docs.google.com/document/d/1QZE5aemQ1lzBN_2RA8-tTVk5JVQ_fNNB4VeSEWKWl8w/edit?usp=sharing). From 52e8ac670907267ec4380b8eb72d32476c1ca82e Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:12:35 -0400 Subject: [PATCH 05/30] Create howtointerpretblockdatafortrades.md --- .../howtointerpretblockdatafortrades.md | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 pages/guides/howtointerpretblockdatafortrades.md diff --git a/pages/guides/howtointerpretblockdatafortrades.md b/pages/guides/howtointerpretblockdatafortrades.md new file mode 100644 index 00000000..ed019dad --- /dev/null +++ b/pages/guides/howtointerpretblockdatafortrades.md @@ -0,0 +1,53 @@ +## dYdX v4: How to Interpret the Block Data for Trades + +Screenshot 2024-03-29 at 2 34 06 PM + +In dYdX trading, quantities and prices are represented in quantums (for quantities) and subticks (for prices), which need conversion for practical understanding. + +### Quantums + +The smallest increment of position size. Determined from `atomicResolution`. + +atomicResolution - Determines the size of a quantum. [For example](https://github.com/dydxprotocol/v4/blob/08069ba905753158b9f390ca52e3f9f0fb2cb3d5/config.yml#L101), an `atomicResolution` of 10 for `BTC`, means that 1 quantum is `1e-10` `BTC`. + +### Subticks + +Human-readable units: `USDC/` e.g. USDC/BTC + +Units in V4 protocol: `quote quantums/base quantums` e.g. (`1e-14 USDC/1e-10 BTC`) + +Determined by `quantum_conversion_exponent`, this allows for flexibility in the case that an asset’s prices plummet, since prices are represent in subticks, decreasing `subticks_per_tick` would allow for ticks to denote smaller increments between prices. + +E.g. 1 `subtick` = `1e-14 USDC/1e-10 BTC` and if BTC was at 20,000 USDC/BTC, a `tick` being 100 USDC/BTC (`subtick_per_tick` = 10000) may make sense. + +If BTC drops to 200 USDC/BTC, a `tick` being 100 USDC/BTC no longer makes sense, and we may want a `tick` to be 1 USDC/BTC, which lets us set `subtick_per_tick` to 100 to get to a `tick` size of 1 USDC/BTC. + +### Now back to the interpretation of the above image: + +1. First, notice column I is negative. That means this trade is a sell by the taker account. If It was positive, it would be a buy. + +Result: Determined if this is a buy or a sell + +2. Next, look at column N. The perpetual_id is 7, which maps to AVAX-USD market. You can see all the mappings from this endpoint https://indexer.dydx.trade/v4/perpetualMarkets where the clobPairId is the perpetual_id. + +Result: Determined the market + +3. Next, we need to get the decimals for this market. First, get the atomicResolution from that endpoint above which we see is -7. Now we can get the size of the trade. From column I and J, take this number -500000000 and multiply by 10^(AtomicResolution) and you get: -500000000 x 10^-7 = 50, so the quantity is 50. + +Result: Determined the quantity + +4. Next, look at columns, E, F, G, H, I, and J + +Screenshot 2024-03-29 at 2 37 28 PM + +The price of the trade is either abs((G+E)/I)*10e(-6 - AtomicResolution), or abs((H+F)/J)*10e(-6 - AtomicResolution), either one is the same. Note that the ‘-6’ is because the AtomicResolution of USDC is -6. + +abs((1479130125 + 369875)/-500000000)*10e(-6 + 7) = 29.59 + +abs((-1479337255 - 162745)/500000000)*10e(-6 +7) = 29.59 + +Result: Determined the price + +### Conclusion + +In conclusion, we have determined that this trade is SELL 50 AVAX-USD at price $29.59 From a707a5260437cd9692ffa23b3865d6409700da17 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:13:22 -0400 Subject: [PATCH 06/30] Rename howtointerpretblockdatafortrades.md to how_to_interpret_block_data_for_trades.md --- ...datafortrades.md => how_to_interpret_block_data_for_trades.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pages/guides/{howtointerpretblockdatafortrades.md => how_to_interpret_block_data_for_trades.md} (100%) diff --git a/pages/guides/howtointerpretblockdatafortrades.md b/pages/guides/how_to_interpret_block_data_for_trades.md similarity index 100% rename from pages/guides/howtointerpretblockdatafortrades.md rename to pages/guides/how_to_interpret_block_data_for_trades.md From 6019d806439f3adc5562d620ac86eb55d23dbf29 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:14:07 -0400 Subject: [PATCH 07/30] Update _meta.json --- pages/guides/_meta.json | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pages/guides/_meta.json b/pages/guides/_meta.json index 3f6b516b..72dcd54a 100644 --- a/pages/guides/_meta.json +++ b/pages/guides/_meta.json @@ -1,6 +1,7 @@ { "how_to_set_up_full_node": "How to set up a full node", "how_to_uncross_orderbook": "How to uncross the orderbook", - "how_to_send_usdc_to_dydx": "How to send USDC from Ethereum to dYdX" + "how_to_send_usdc_to_dydx": "How to send USDC from Ethereum to dYdX", + "how_to_interpret_block_data_for_trades": "How to interpret block data for trades" } - \ No newline at end of file + From 0ee43805bd449c5e49a71e0a9bf96cb272288952 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:15:14 -0400 Subject: [PATCH 08/30] Create _meta.json --- pages/Architecture/_meta.json | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 pages/Architecture/_meta.json diff --git a/pages/Architecture/_meta.json b/pages/Architecture/_meta.json new file mode 100644 index 00000000..5095bf78 --- /dev/null +++ b/pages/Architecture/_meta.json @@ -0,0 +1,4 @@ +{ + "architectural_overview": "Architectural Overview", + "indexer": "Indexer" + } From 7108e6a8475b5c3aef9e0bdbbc34cf78b683dd6c Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:15:30 -0400 Subject: [PATCH 09/30] Rename ArchitecturalOverview.md to architectural_overview.md --- .../{ArchitecturalOverview.md => architectural_overview.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pages/Architecture/{ArchitecturalOverview.md => architectural_overview.md} (100%) diff --git a/pages/Architecture/ArchitecturalOverview.md b/pages/Architecture/architectural_overview.md similarity index 100% rename from pages/Architecture/ArchitecturalOverview.md rename to pages/Architecture/architectural_overview.md From ff46c90dbe73ef506869dd440ddc838222bb9778 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:15:43 -0400 Subject: [PATCH 10/30] Rename Indexer.md to indexer.md --- pages/Architecture/{Indexer.md => indexer.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pages/Architecture/{Indexer.md => indexer.md} (100%) diff --git a/pages/Architecture/Indexer.md b/pages/Architecture/indexer.md similarity index 100% rename from pages/Architecture/Indexer.md rename to pages/Architecture/indexer.md From 0e35a7b781dcdd432a06dec7762c09e3bcc9fb12 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:29:55 -0400 Subject: [PATCH 11/30] Create maintaining_a_local_orderbook.md --- pages/guides/maintaining_a_local_orderbook.md | 99 +++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 pages/guides/maintaining_a_local_orderbook.md diff --git a/pages/guides/maintaining_a_local_orderbook.md b/pages/guides/maintaining_a_local_orderbook.md new file mode 100644 index 00000000..edf710b0 --- /dev/null +++ b/pages/guides/maintaining_a_local_orderbook.md @@ -0,0 +1,99 @@ +# Maintaining a local orderbook + +# NEED JAY TO ADD SHORT INTRO + +Building a local orderbook should be fairly straight forward. Here is a quick [example PR](https://github.com/dydxprotocol/v4-chain/pull/1268) for a Go GRPC client that subscribes to the orderbook updates and maintains an orderbook locally. + +Specifically after subscribing to the orderbook updates: + +- Use the orderbook in the snapshot as the starting orderbook. +- Add the corresponding order to the end of the price level when `OrderPlaceV1` is received. + +```go +func (l *LocalOrderbook) AddOrder(order v1types.IndexerOrder) { + l.Lock() + defer l.Unlock() + + if _, ok := l.OrderIdToOrder[order.OrderId]; ok { + l.Logger.Error("order already exists in orderbook") + } + + subticks := order.GetSubticks() + if order.Side == v1types.IndexerOrder_SIDE_BUY { + if _, ok := l.Bids[subticks]; !ok { + l.Bids[subticks] = make([]v1types.IndexerOrder, 0) + } + l.Bids[subticks] = append(l.Bids[subticks], order) + } else { + if _, ok := l.Asks[subticks]; !ok { + l.Asks[subticks] = make([]v1types.IndexerOrder, 0) + } + l.Asks[subticks] = append(l.Asks[subticks], order) + } + + l.OrderIdToOrder[order.OrderId] = order + l.OrderRemainingAmount[order.OrderId] = 0 +} +``` + +- Update the order remaining size when `OrderUpdateV1` is received + +```go +func (l *LocalOrderbook) SetOrderRemainingAmount(orderId v1types.IndexerOrderId, totalFilledQuantums uint64) { + l.Lock() + defer l.Unlock() + + order := l.OrderIdToOrder[orderId] + if totalFilledQuantums > order.Quantums { + l.Logger.Error("totalFilledQuantums > order.Quantums") + } + l.OrderRemainingAmount[orderId] = order.Quantums - totalFilledQuantums +} +``` + +- Remove the order from the orderbook when `OrderRemoveV1` is received. + +```go +func (l *LocalOrderbook) RemoveOrder(orderId v1types.IndexerOrderId) { + l.Lock() + defer l.Unlock() + + if _, ok := l.OrderIdToOrder[orderId]; !ok { + l.Logger.Error("order not found in orderbook") + } + + order := l.OrderIdToOrder[orderId] + subticks := order.GetSubticks() + + if order.Side == v1types.IndexerOrder_SIDE_BUY { + for i, o := range l.Bids[subticks] { + if o.OrderId == order.OrderId { + l.Bids[subticks] = append( + l.Bids[subticks][:i], + l.Bids[subticks][i+1:]..., + ) + break + } + } + if len(l.Bids[subticks]) == 0 { + delete(l.Bids, subticks) + } + } else { + for i, o := range l.Asks[subticks] { + if o.OrderId == order.OrderId { + l.Asks[subticks] = append( + l.Asks[subticks][:i], + l.Asks[subticks][i+1:]..., + ) + break + } + } + if len(l.Asks[subticks]) == 0 { + delete(l.Asks, subticks) + } + } + + delete(l.OrderRemainingAmount, orderId) + delete(l.OrderIdToOrder, orderId) +} +``` From 70ec1e69b88f19eec39b05b616f9a5fd071c111c Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:33:55 -0400 Subject: [PATCH 12/30] Create using_Cosmovisor_to_stage_dYdXChain.md --- .../using_Cosmovisor_to_stage_dYdXChain.md | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 pages/guides/using_Cosmovisor_to_stage_dYdXChain.md diff --git a/pages/guides/using_Cosmovisor_to_stage_dYdXChain.md b/pages/guides/using_Cosmovisor_to_stage_dYdXChain.md new file mode 100644 index 00000000..14489776 --- /dev/null +++ b/pages/guides/using_Cosmovisor_to_stage_dYdXChain.md @@ -0,0 +1,39 @@ +# Using Cosmovisor to stage dYdXChain binary upgrade + +## Prerequisite + +1. Linux (Ubuntu Server 22.04.3 recommended) +2. 8-cpu (ARM or x86_64), 64 GB RAM, 500 GB SSD NVME Storage +3. Already installed dYdXChain full node + +## Preparation + +1. Install Go from https://go.dev/doc/install (Version tested is 1.22.1) +2. Install Cosmovisor, with the following command: +go install cosmossdk.io/tools/cosmovisor/cmd/cosmovisor@latest +3. Copy cosmovisor from $HOME/go/bin/ to a directory in your $PATH +4. Add two environment variables to $HOME/.profile. The data directory is typically $HOME/.dydx-mainnet-1 +export DAEMON_NAME=dydxprotocold +export DAEMON_HOME= +5. Log out and log back in. +6. Initialize Cosmovisor with the following command. The is the the full path to dydxprotocold +cosmovisor init +7.Cosmovisor is now ready for use. + +## Running dydxprotocold under Cosmovisor + +You have to change the way you currently run dydxprotocold to run under Cosmovisor. This is done simply by specifying “cosmovisor run” in place of the “dydxprotocold” command you used previously. Therefore, if you previously used “dydxprotocold start --p2p.seeds="ade4d8…”, you would change that to “cosmovisor run start --p2p.seeds="ade4d8…” + +## Staging upgrade + +1. The Cosmovisor directory structure looks like this: + +Screenshot 2024-04-01 at 7 31 42 PM + +2. To stage an upgrade, you would create a directory inside the upgrades/ directory. For example, as of 4/1/2024, the current version is v3.0.0 and the next upgrade version is v4.0.0. Therefore you would create a directory called “v4.0.0” and then a bin directory inside it. + +Screenshot 2024-04-01 at 7 32 11 PM + +3. Now, download the upgraded binary and put it inside the bin directory created previously. It must be named dydxprotocold + +4. Restart dydxprotocold with Cosmovisor. Now, Cosmovisor will automatically halt the current binary at the block activation height and start the upgrade binary. From d9021d7898b196c2c05ac84dec079dfa6d314e32 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:35:11 -0400 Subject: [PATCH 13/30] Update _meta.json --- pages/guides/_meta.json | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/pages/guides/_meta.json b/pages/guides/_meta.json index 72dcd54a..c8ef4bfb 100644 --- a/pages/guides/_meta.json +++ b/pages/guides/_meta.json @@ -2,6 +2,8 @@ "how_to_set_up_full_node": "How to set up a full node", "how_to_uncross_orderbook": "How to uncross the orderbook", "how_to_send_usdc_to_dydx": "How to send USDC from Ethereum to dYdX", - "how_to_interpret_block_data_for_trades": "How to interpret block data for trades" + "how_to_interpret_block_data_for_trades": "How to interpret block data for trades", + "maintaining_a_local_orderbook": "Maintaining a Local Orderbook", + "using_Cosmovisor_to_stage_dYdXChain": "Using Cosmovisor to stage dYdX Chain binary" } From 247048d42f99a4672c59fc79ed2de8fac1841676 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 19:40:00 -0400 Subject: [PATCH 14/30] Update maintaining_a_local_orderbook.md --- pages/guides/maintaining_a_local_orderbook.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/pages/guides/maintaining_a_local_orderbook.md b/pages/guides/maintaining_a_local_orderbook.md index edf710b0..eddbeb3f 100644 --- a/pages/guides/maintaining_a_local_orderbook.md +++ b/pages/guides/maintaining_a_local_orderbook.md @@ -1,7 +1,5 @@ # Maintaining a local orderbook -# NEED JAY TO ADD SHORT INTRO - Building a local orderbook should be fairly straight forward. Here is a quick [example PR](https://github.com/dydxprotocol/v4-chain/pull/1268) for a Go GRPC client that subscribes to the orderbook updates and maintains an orderbook locally. Specifically after subscribing to the orderbook updates: From beb9e2425d3a75b56990cbd9e4a2d2577e743770 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Mon, 1 Apr 2024 20:54:44 -0400 Subject: [PATCH 15/30] Update _meta.json --- pages/_meta.json | 1 + 1 file changed, 1 insertion(+) diff --git a/pages/_meta.json b/pages/_meta.json index 4b634bb3..f49618c1 100644 --- a/pages/_meta.json +++ b/pages/_meta.json @@ -12,6 +12,7 @@ "governance": "Governance", "security": "Security", "guides":"Guides", + "FAQ": "FAQs", "terms_and_policies": "Terms and Policies", "app": { "title": "Testnet ↗", From 46a934975070547bb78d169bd7ad83aa2858a7ba Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 11:53:16 -0400 Subject: [PATCH 16/30] Update maintaining_a_local_orderbook.md --- pages/guides/maintaining_a_local_orderbook.md | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/pages/guides/maintaining_a_local_orderbook.md b/pages/guides/maintaining_a_local_orderbook.md index eddbeb3f..434a5886 100644 --- a/pages/guides/maintaining_a_local_orderbook.md +++ b/pages/guides/maintaining_a_local_orderbook.md @@ -1,5 +1,53 @@ # Maintaining a local orderbook +## Orderbook Stream + +This feature aims to provide real-time and accurate orderbook updates. Complete orderbook activities are streamed to the client and can be used to construct a full depth L3 orderbook. Streams are implemented using the existing GRPC query service from Cosmos SDK. + +The initial implementation only contains orders but not trades. Also note that by dYdX V4’s design, the orderbook can be slightly different across different nodes. + +## Enabling GRPC Streaming + +This feature can be enabled via a command line flag (--grpc-streaming-enabled=true) when starting your full node. This feature can only be used on non validating full nodes and when grpc is also enabled. + +## Request / Response + +To subscribe to the stream, the client can send a 'StreamOrderbookUpdatesRequest' specifying the clob pair ids to subscribe to. + +```go +// StreamOrderbookUpdatesRequest is a request message for the +// StreamOrderbookUpdates method. +message StreamOrderbookUpdatesRequest { + // Clob pair ids to stream orderbook updates for. + repeated uint32 clob_pair_id = 1; +} +``` + +Response will contain the orderbook updates (Add/Remove/Update), whether the updates are coming from a snapshot, and a few fields used for debugging issues. + +```go +// StreamOrderbookUpdatesResponse is a response message for the +// StreamOrderbookUpdates method. +message StreamOrderbookUpdatesResponse { + // Orderbook updates for the clob pair. + repeated dydxprotocol.indexer.off_chain_updates.OffChainUpdateV1 updates = 1 + [ (gogoproto.nullable) = false ]; + + // Snapshot indicates if the response is from a snapshot of the orderbook. + // This is true for the initial response and false for all subsequent updates. + // Note that if the snapshot is true, then all previous entries should be + // discarded and the orderbook should be resynced. + bool snapshot = 2; + + // ---Additional fields used to debug issues--- + // Block height of the updates. + uint32 block_height = 3; + + // Exec mode of the updates. + uint32 exec_mode = 4; +} +``` + Building a local orderbook should be fairly straight forward. Here is a quick [example PR](https://github.com/dydxprotocol/v4-chain/pull/1268) for a Go GRPC client that subscribes to the orderbook updates and maintains an orderbook locally. Specifically after subscribing to the orderbook updates: From b2e47af88b3db171efe46f726f774c480193a490 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 11:54:24 -0400 Subject: [PATCH 17/30] Update and rename maintaining_a_local_orderbook.md to orderbook_stream.md --- ...maintaining_a_local_orderbook.md => orderbook_stream.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename pages/guides/{maintaining_a_local_orderbook.md => orderbook_stream.md} (98%) diff --git a/pages/guides/maintaining_a_local_orderbook.md b/pages/guides/orderbook_stream.md similarity index 98% rename from pages/guides/maintaining_a_local_orderbook.md rename to pages/guides/orderbook_stream.md index 434a5886..1599b72d 100644 --- a/pages/guides/maintaining_a_local_orderbook.md +++ b/pages/guides/orderbook_stream.md @@ -1,6 +1,4 @@ -# Maintaining a local orderbook - -## Orderbook Stream +# Orderbook Stream This feature aims to provide real-time and accurate orderbook updates. Complete orderbook activities are streamed to the client and can be used to construct a full depth L3 orderbook. Streams are implemented using the existing GRPC query service from Cosmos SDK. @@ -48,6 +46,8 @@ message StreamOrderbookUpdatesResponse { } ``` +## Maintaining a local orderbook + Building a local orderbook should be fairly straight forward. Here is a quick [example PR](https://github.com/dydxprotocol/v4-chain/pull/1268) for a Go GRPC client that subscribes to the orderbook updates and maintains an orderbook locally. Specifically after subscribing to the orderbook updates: From 2ac259bcce824fcb09612362bb5f54115a1cf289 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 11:54:50 -0400 Subject: [PATCH 18/30] Update _meta.json --- pages/guides/_meta.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/guides/_meta.json b/pages/guides/_meta.json index c8ef4bfb..1b21c30f 100644 --- a/pages/guides/_meta.json +++ b/pages/guides/_meta.json @@ -3,7 +3,7 @@ "how_to_uncross_orderbook": "How to uncross the orderbook", "how_to_send_usdc_to_dydx": "How to send USDC from Ethereum to dYdX", "how_to_interpret_block_data_for_trades": "How to interpret block data for trades", - "maintaining_a_local_orderbook": "Maintaining a Local Orderbook", + "orderbook_stream": "Orderbook Stream", "using_Cosmovisor_to_stage_dYdXChain": "Using Cosmovisor to stage dYdX Chain binary" } From 57096269c9d5c02a3f068ec34e57eb892ce0adc3 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 16:24:26 -0400 Subject: [PATCH 19/30] Update orderbook_stream.md --- pages/guides/orderbook_stream.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/pages/guides/orderbook_stream.md b/pages/guides/orderbook_stream.md index 1599b72d..e20a8abd 100644 --- a/pages/guides/orderbook_stream.md +++ b/pages/guides/orderbook_stream.md @@ -46,6 +46,21 @@ message StreamOrderbookUpdatesResponse { } ``` +## Example Scenario + +- Trader places a bid at price 100 for size 1 + - OrderPlace, price = 100, size = 1 + - OrderUpdate, total filled amount = 0 +- Trader replaces that original bid to be price 99 at size 2 + - OrderRemove + - OrderPlace, price = 99, size = 2 + - OrderUpdate, total filled amount = 0 +- Another trader submits an IOC ask at price 100 for size 1. + - Full node doesn't see this matching anything so no updates. +- Block is confirmed that there was a fill for the trader's original order at price 100 for size 1 (BP didn't see the order replacement) + - OrderUpdate, total filled amount = 1 + + ## Maintaining a local orderbook Building a local orderbook should be fairly straight forward. Here is a quick [example PR](https://github.com/dydxprotocol/v4-chain/pull/1268) for a Go GRPC client that subscribes to the orderbook updates and maintains an orderbook locally. From b5f01af39a8419ae713f2a4120f6c3a751173dd7 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 16:28:45 -0400 Subject: [PATCH 20/30] Rename _meta.json to architecture.json --- pages/Architecture/{_meta.json => architecture.json} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pages/Architecture/{_meta.json => architecture.json} (100%) diff --git a/pages/Architecture/_meta.json b/pages/Architecture/architecture.json similarity index 100% rename from pages/Architecture/_meta.json rename to pages/Architecture/architecture.json From 15d4c9a21e293f1e21ad1da49a200107bb89b87d Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 16:33:56 -0400 Subject: [PATCH 21/30] Rename _meta.json to developers.json --- pages/developers/{_meta.json => developers.json} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pages/developers/{_meta.json => developers.json} (100%) diff --git a/pages/developers/_meta.json b/pages/developers/developers.json similarity index 100% rename from pages/developers/_meta.json rename to pages/developers/developers.json From 33ba80197e2c0a6375daf1f3563260655d037885 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 16:34:16 -0400 Subject: [PATCH 22/30] Rename _meta.json to deposits_and_withdrawals.json --- .../{_meta.json => deposits_and_withdrawals.json} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pages/deposits_and_withdrawals/{_meta.json => deposits_and_withdrawals.json} (100%) diff --git a/pages/deposits_and_withdrawals/_meta.json b/pages/deposits_and_withdrawals/deposits_and_withdrawals.json similarity index 100% rename from pages/deposits_and_withdrawals/_meta.json rename to pages/deposits_and_withdrawals/deposits_and_withdrawals.json From 7cac00d9495d6436bcc9586c999390c9fb0eddc2 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 16:37:51 -0400 Subject: [PATCH 23/30] Rename _meta.json to clients.json --- pages/developers/clients/{_meta.json => clients.json} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pages/developers/clients/{_meta.json => clients.json} (100%) diff --git a/pages/developers/clients/_meta.json b/pages/developers/clients/clients.json similarity index 100% rename from pages/developers/clients/_meta.json rename to pages/developers/clients/clients.json From 298151ee44317ce361726d3b354828461f7fa213 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 16:51:10 -0400 Subject: [PATCH 24/30] Update architectural_overview.md --- pages/Architecture/architectural_overview.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/pages/Architecture/architectural_overview.md b/pages/Architecture/architectural_overview.md index bf9b824f..8bb7897c 100644 --- a/pages/Architecture/architectural_overview.md +++ b/pages/Architecture/architectural_overview.md @@ -2,17 +2,16 @@ v4 (or dYdX Chain) is the next iteration of the dYdX protocol, which will consist of open-source software. The current version live in production today is known as v3 and can be found [here](https://trade.dydx.exchange/?utm_source=dydx-website&utm_medium=blog&utm_content=v4-technical-architecture-overview). v3 and past versions of dYdX were, at their core, smart contracts deployed to existing chains, combined with centralized services hosted in the cloud. v4 will be a standalone L1 blockchain, featuring a fully decentralized, off-chain orderbook and matching engine. The dYdX Chain will be based on the Cosmos SDK and CometBFT PoS consensus protocol. -As we approach the release of v4 mainnet open-source software (“dYdX Chain”), we want to provide a peek into what the dYdX team is building. This post presents a high-level overview of the architecture of v4. Given that v4 is still under development, please note that what is presented below is subject to change. - ### v4 System Architecture -dYdX v4 is being designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components will be made available as open source software. None of the components will be run by dYdX Trading Inc. + +dYdX v4 is being designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components are available as open source software. None of the components are run by dYdX Trading Inc. ![image](https://github.com/dydxprotocol/v4-documentation/assets/130097657/e9a54253-e7fa-44ab-97c5-ae1ce7cae320) ### Protocol (or “Application”) The open-source protocol is an L1 blockchain built on top of [CometBFT](https://dydx.exchange/blog/v4-technical-architecture-overview#:~:text=on%20top%20of-,CometBFT,-and%20using%20CosmosSDK) and using [CosmosSDK](https://v1.cosmos.network/sdk). The node software is written in Go, and compiles to a single binary. Like all CosmosSDK blockchains, v4 uses a proof-of-stake consensus mechanism. -The protocol will be supported by a network of nodes. There are two types of nodes: +The protocol is supported by a network of nodes. There are two types of nodes: - **Validators**: Validators are responsible for storing orders in an in-memory orderbook (i.e. off chain and not committed to consensus), gossipping transactions to other validators, and producing new blocks for the dYdX Chain through the consensus process. The consensus process will have validators take turns as the proposer of new blocks in a weighted-round-robin fashion (weighted by the number of tokens staked to their node). The proposer is responsible for proposing the contents of the next block. When an order gets matched, the proposer adds it to their proposed block and initiates a consensus round. If ⅔ or more of the validators (by stake weight) approve of a block, then the block is considered committed and added to the blockchain. Users will submit transactions directly to validators. @@ -23,17 +22,17 @@ The Indexer is a read-only collection of services whose purpose is to index and While the v4 open-source protocol itself is capable of exposing endpoints to service queries about some basic on-chain data, those queries tend to be slow as validators and full nodes are not optimized to efficiently handle them. Additionally, an excess of queries to a validator can impair its ability to participate in consensus. For this reason, many Cosmos validators tend to disable these APIs in production. This is why it is important to build and maintain Indexer and full-node software separate from validator software. -Indexers will use Postgres databases to store on-chain data, Redis for off-chain data, and Kafka for consuming and streaming on/off-chain data to the various Indexer services. +Indexers use Postgres databases to store on-chain data, Redis for off-chain data, and Kafka for consuming and streaming on/off-chain data to the various Indexer services. ### Front-ends -In service of building an end-to-end decentralized experience, dYdX is building three open-source front ends: a web app, an iOS app, and an Android app. +In service of building an end-to-end decentralized experience, dYdX has built three open-source front ends: a web app, an iOS app, and an Android app. -- **Web application**: The website will be built using Javascript and React. The website will interact with the Indexer through an API to get off-chain orderbook information and will send trades directly to the chain. dYdX will open source the front end codebase and associated deployment scripts. This will allow for anyone to easily deploy and access the dYdX front end to/from their own domain/hosting solution via IPFS/Cloudflare gateway. +- **Web application**: The website was built using Javascript and React. The website interacts with the Indexer through an API to get off-chain orderbook information and will send trades directly to the chain. dYdX has open sourced the front end codebase and associated deployment scripts. This allows anyone to easily deploy and access the dYdX front end to/from their own domain/hosting solution via IPFS/Cloudflare gateway. -- **Mobile**: The iOS and Android apps are built in native Swift and Kotlin, respectively. The mobile apps will interact with the Indexer in the same way the web application does, and will send trades directly to the chain. The mobile apps will be open sourced as well, allowing anyone to deploy the mobile app to the App Store or Play store. Specifically for the App store, the deployer will need to have a developer account as well as a Bitrise account to go through the app submission process. +- **Mobile**: The iOS and Android apps are built in native Swift and Kotlin, respectively. The mobile apps interact with the Indexer in the same way the web application does, and will send trades directly to the chain. The mobile apps have been open sourced as well, allowing anyone to deploy the mobile app to the App Store or Play store. Specifically for the App store, the deployer needs to have a developer account as well as a Bitrise account to go through the app submission process. ### Lifecycle of an Order -Now that we have a better understanding of each of the components of dYdX v4, let’s take a look at how it all comes together when placing an order. When an order is placed on v4, it will follow the flow below: +Now that we have a better understanding of each of the components of dYdX v4, let’s take a look at how it all comes together when placing an order. When an order is placed on v4, it follows the flow below: 1. User places a trade on a decentralized front-end (e.g., website) or via API 2. The order is routed to a validator. That validator gossips that transaction to other validators and full nodes to update their orderbooks with the new order. @@ -42,5 +41,3 @@ Now that we have a better understanding of each of the components of dYdX v4, le 1. If ⅔ of validator nodes vote to confirm the block, then the block is committed and saved to the on-chain databases of all validators and full nodes. 2. If the proposed block does not successfully hit the ⅔ threshold, then the block is rejected. 5. After the block is committed, the updated on-chain (and off-chain) data is streamed from full nodes to Indexers. The Indexer then makes this data available via API and Websockets back to the front end and/or any other outside services querying for this data. - -The flow above is a high level overview of how an order/data will move through v4. We will do further deep dives into the protocol, indexer, and various front ends’ infrastructure in subsequent blog posts as we get closer to the release of v4 mainnet open-source software. From e14224d320dad7471df8c95f7359ed08aa68b236 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 16:58:17 -0400 Subject: [PATCH 25/30] Update indexer.md --- pages/Architecture/indexer.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pages/Architecture/indexer.md b/pages/Architecture/indexer.md index 4b8f2a3f..1eafabd2 100644 --- a/pages/Architecture/indexer.md +++ b/pages/Architecture/indexer.md @@ -1,10 +1,10 @@ # Indexer Deep Dive -A good way to think about the indexer is as similar to Infura or Alchemy’s role in the Ethereum ecosystem. However, unlike Infura/Alchemy, and like everything else in dYdX v4, the Indexer is completely open source and can be run by anyone! +A good way to think about the indexer is as similar to Infura or Alchemy’s role in the Ethereum ecosystem. However, unlike Infura/Alchemy, and like everything else in dYdX Chain, the Indexer is completely open source and can be run by anyone! ### What is the Indexer? -As part of tooling for the dYdX ecosystem, we want to ensure that clients have access to performant data queries when using the dYdX v4 exchange. Cosmos SDK Full Nodes offer a number of APIs that can be used to request on-chain data. However, these Full Nodes are optimized for committing and executing blocks, not for serving high frequency, low-latency requests from web/mobile clients. +As part of tooling for the dYdX ecosystem, we want to ensure that clients have access to performant data queries when using exchanges running on dYdX Chain software. Cosmos SDK Full Nodes offer a number of APIs that can be used to request on-chain data. However, these Full Nodes are optimized for committing and executing blocks, not for serving high frequency, low-latency requests from web/mobile clients. This is why we wrote software for an indexing service. The Indexer is a read-only service that serves off chain data to clients over REST APIs and Websockets. Its purpose is to store and serve data that exists on the dYdX Chain in an easier to use way. In other words, the purpose of an indexer is to index and serve data to clients in a more performant, efficient and web2-friendly way. For example the indexer will serve websockets that provide updates on the state of the orderbook and fills. These clients will include front-end applications (mobile and web), market makers, institutions, and any other parties looking to query dYdX Chain data via a traditional web2 API. @@ -55,7 +55,7 @@ Vulcan is the Indexer’s off-chain data ingestion service. It will consume data Comlink is an API server that will expose REST API endpoints to read both on-chain and off-chain data. For example, a user could request their USDC balance or the size of a particular position through Comlink, and would receive a formatted JSON response. -As an explicit goal set out by the dYdX team, we’re designing v4 APIs to closely match the [v3 exchange APIs](https://dydx.exchange/blog/v4-deep-dive-indexer#:~:text=closely%20match%20the-,v3%20exchange%20APIs,-.%20We%20have%20had). We have had time to gather feedback and iterate on these APIs over time with v3, and have confidence that they are reasonable at the product-level. +As an explicit goal set out by the dYdX team, we’re designing v4 APIs to closely match the [v3 APIs](https://dydx.exchange/blog/v4-deep-dive-indexer#:~:text=closely%20match%20the-,v3%20exchange%20APIs,-.%20We%20have%20had). We have had time to gather feedback and iterate on these APIs over time with v3, and have confidence that they are reasonable at the product-level. ### Roundtable From 0195293552a54ad582f2c140626c8dacfac7d231 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 17:01:45 -0400 Subject: [PATCH 26/30] Update FAQ.md --- pages/FAQ.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/FAQ.md b/pages/FAQ.md index 8c40adb2..1e6f733a 100644 --- a/pages/FAQ.md +++ b/pages/FAQ.md @@ -1,6 +1,6 @@ # FAQ Documentation -Since the inception of v4, we have been compiling a list of relevant questions and answers that are asked by MMs/Trading Firms on Telegram. We turned this list into a live _v4 Telegram Questions_ document, which should be able to answer many of your more nuanced questions. +Since the inception of dYdX Chain, we have been compiling a list of questions and answers from users related to technical support. We turned this list into a live _v4 Telegram Questions_ document, which should be able to answer many of your more nuanced questions. It's a large document, so we recommend CTRL+F to find key words. From f35b19a8466934fef519af9c5ef4cca384d97a12 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 17:03:30 -0400 Subject: [PATCH 27/30] Update how_to_interpret_block_data_for_trades.md --- pages/guides/how_to_interpret_block_data_for_trades.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pages/guides/how_to_interpret_block_data_for_trades.md b/pages/guides/how_to_interpret_block_data_for_trades.md index ed019dad..f38c4212 100644 --- a/pages/guides/how_to_interpret_block_data_for_trades.md +++ b/pages/guides/how_to_interpret_block_data_for_trades.md @@ -2,7 +2,7 @@ Screenshot 2024-03-29 at 2 34 06 PM -In dYdX trading, quantities and prices are represented in quantums (for quantities) and subticks (for prices), which need conversion for practical understanding. +In dYdX Chain trading, quantities and prices are represented in quantums (for quantities) and subticks (for prices), which need conversion for practical understanding. ### Quantums @@ -28,7 +28,7 @@ If BTC drops to 200 USDC/BTC, a `tick` being 100 USDC/BTC no longer makes sense, Result: Determined if this is a buy or a sell -2. Next, look at column N. The perpetual_id is 7, which maps to AVAX-USD market. You can see all the mappings from this endpoint https://indexer.dydx.trade/v4/perpetualMarkets where the clobPairId is the perpetual_id. +2. Next, look at column N. The perpetual_id is 7, which maps to AVAX-USD market. You can see all the mappings from this endpoint for the dYdX Chain deployment by dYdX Operations Services Ltd. https://indexer.dydx.trade/v4/perpetualMarkets where the clobPairId is the perpetual_id. Result: Determined the market From 0f645dbfcc61d73635bbeaa90829449263ed5f94 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 17:08:52 -0400 Subject: [PATCH 28/30] Update architectural_overview.md --- pages/Architecture/architectural_overview.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/pages/Architecture/architectural_overview.md b/pages/Architecture/architectural_overview.md index 8bb7897c..293d8c60 100644 --- a/pages/Architecture/architectural_overview.md +++ b/pages/Architecture/architectural_overview.md @@ -1,14 +1,13 @@ # Intro to dYdX v4 Architecture -v4 (or dYdX Chain) is the next iteration of the dYdX protocol, which will consist of open-source software. The current version live in production today is known as v3 and can be found [here](https://trade.dydx.exchange/?utm_source=dydx-website&utm_medium=blog&utm_content=v4-technical-architecture-overview). v3 and past versions of dYdX were, at their core, smart contracts deployed to existing chains, combined with centralized services hosted in the cloud. v4 will be a standalone L1 blockchain, featuring a fully decentralized, off-chain orderbook and matching engine. The dYdX Chain will be based on the Cosmos SDK and CometBFT PoS consensus protocol. - ### v4 System Architecture -dYdX v4 is being designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components are available as open source software. None of the components are run by dYdX Trading Inc. +dYdX Chain has been designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components are available as open source software. None of the components are run by dYdX Trading Inc. ![image](https://github.com/dydxprotocol/v4-documentation/assets/130097657/e9a54253-e7fa-44ab-97c5-ae1ce7cae320) ### Protocol (or “Application”) + The open-source protocol is an L1 blockchain built on top of [CometBFT](https://dydx.exchange/blog/v4-technical-architecture-overview#:~:text=on%20top%20of-,CometBFT,-and%20using%20CosmosSDK) and using [CosmosSDK](https://v1.cosmos.network/sdk). The node software is written in Go, and compiles to a single binary. Like all CosmosSDK blockchains, v4 uses a proof-of-stake consensus mechanism. The protocol is supported by a network of nodes. There are two types of nodes: From ca97d12ee2437875b3a7e25500ffd684d9fb7439 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 17:09:21 -0400 Subject: [PATCH 29/30] Update architectural_overview.md --- pages/Architecture/architectural_overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/Architecture/architectural_overview.md b/pages/Architecture/architectural_overview.md index 293d8c60..463615fd 100644 --- a/pages/Architecture/architectural_overview.md +++ b/pages/Architecture/architectural_overview.md @@ -31,7 +31,7 @@ In service of building an end-to-end decentralized experience, dYdX has built th - **Mobile**: The iOS and Android apps are built in native Swift and Kotlin, respectively. The mobile apps interact with the Indexer in the same way the web application does, and will send trades directly to the chain. The mobile apps have been open sourced as well, allowing anyone to deploy the mobile app to the App Store or Play store. Specifically for the App store, the deployer needs to have a developer account as well as a Bitrise account to go through the app submission process. ### Lifecycle of an Order -Now that we have a better understanding of each of the components of dYdX v4, let’s take a look at how it all comes together when placing an order. When an order is placed on v4, it follows the flow below: +Now that we have a better understanding of each of the components of dYdX Chain, let’s take a look at how it all comes together when placing an order. When an order is placed on v4, it follows the flow below: 1. User places a trade on a decentralized front-end (e.g., website) or via API 2. The order is routed to a validator. That validator gossips that transaction to other validators and full nodes to update their orderbooks with the new order. From f04799c3689fb302b812e2105f7d39874f51ef25 Mon Sep 17 00:00:00 2001 From: TommyGarch <130097657+TommyGarch@users.noreply.github.com> Date: Tue, 2 Apr 2024 17:29:32 -0400 Subject: [PATCH 30/30] Update architectural_overview.md --- pages/Architecture/architectural_overview.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/pages/Architecture/architectural_overview.md b/pages/Architecture/architectural_overview.md index 463615fd..f86f3bde 100644 --- a/pages/Architecture/architectural_overview.md +++ b/pages/Architecture/architectural_overview.md @@ -1,25 +1,25 @@ -# Intro to dYdX v4 Architecture +# Intro to dYdX Chain Architecture -### v4 System Architecture +### System Architecture -dYdX Chain has been designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components are available as open source software. None of the components are run by dYdX Trading Inc. +dYdX Chain (sometimes referred to as "v4") has been designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components are available as open source software. None of the components are run by dYdX Trading Inc. ![image](https://github.com/dydxprotocol/v4-documentation/assets/130097657/e9a54253-e7fa-44ab-97c5-ae1ce7cae320) ### Protocol (or “Application”) -The open-source protocol is an L1 blockchain built on top of [CometBFT](https://dydx.exchange/blog/v4-technical-architecture-overview#:~:text=on%20top%20of-,CometBFT,-and%20using%20CosmosSDK) and using [CosmosSDK](https://v1.cosmos.network/sdk). The node software is written in Go, and compiles to a single binary. Like all CosmosSDK blockchains, v4 uses a proof-of-stake consensus mechanism. +The open-source protocol is an L1 blockchain built on top of [CometBFT](https://dydx.exchange/blog/v4-technical-architecture-overview#:~:text=on%20top%20of-,CometBFT,-and%20using%20CosmosSDK) and using [CosmosSDK](https://v1.cosmos.network/sdk). The node software is written in Go, and compiles to a single binary. Like all CosmosSDK blockchains, dYdX Chain uses a proof-of-stake consensus mechanism. The protocol is supported by a network of nodes. There are two types of nodes: - **Validators**: Validators are responsible for storing orders in an in-memory orderbook (i.e. off chain and not committed to consensus), gossipping transactions to other validators, and producing new blocks for the dYdX Chain through the consensus process. The consensus process will have validators take turns as the proposer of new blocks in a weighted-round-robin fashion (weighted by the number of tokens staked to their node). The proposer is responsible for proposing the contents of the next block. When an order gets matched, the proposer adds it to their proposed block and initiates a consensus round. If ⅔ or more of the validators (by stake weight) approve of a block, then the block is considered committed and added to the blockchain. Users will submit transactions directly to validators. -- **Full Nodes**: A Full Node represents a process running the v4 open-source application that does not participate in consensus. It is a node with 0 stake weight and it does not submit proposals or vote on them. However, full nodes are connected to the network of validators, participate in the gossiping of transactions, and also process each new committed block. Full nodes have a complete view of the dYdX Chain and its history, and are intended to support the Indexer. Some parties may decide (either for performance or cost reasons) to run their own full node and/or Indexer. +- **Full Nodes**: A Full Node represents a process running the dYdX Chain open-source application that does not participate in consensus. It is a node with 0 stake weight and it does not submit proposals or vote on them. However, full nodes are connected to the network of validators, participate in the gossiping of transactions, and also process each new committed block. Full nodes have a complete view of the dYdX Chain and its history, and are intended to support the Indexer. Some parties may decide (either for performance or cost reasons) to run their own full node and/or Indexer. ### Indexer -The Indexer is a read-only collection of services whose purpose is to index and serve blockchain data to end users in a more efficient and web2-friendly way. This is done by consuming real time data from a v4 full node, storing it in a database, and serving that data through a websocket and REST requests to end-users. +The Indexer is a read-only collection of services whose purpose is to index and serve blockchain data to end users in a more efficient and web2-friendly way. This is done by consuming real time data from a dYdX Chain full node, storing it in a database, and serving that data through a websocket and REST requests to end-users. -While the v4 open-source protocol itself is capable of exposing endpoints to service queries about some basic on-chain data, those queries tend to be slow as validators and full nodes are not optimized to efficiently handle them. Additionally, an excess of queries to a validator can impair its ability to participate in consensus. For this reason, many Cosmos validators tend to disable these APIs in production. This is why it is important to build and maintain Indexer and full-node software separate from validator software. +While the dYdX Chain open-source protocol itself is capable of exposing endpoints to service queries about some basic on-chain data, those queries tend to be slow as validators and full nodes are not optimized to efficiently handle them. Additionally, an excess of queries to a validator can impair its ability to participate in consensus. For this reason, many Cosmos validators tend to disable these APIs in production. This is why it is important to build and maintain Indexer and full-node software separate from validator software. Indexers use Postgres databases to store on-chain data, Redis for off-chain data, and Kafka for consuming and streaming on/off-chain data to the various Indexer services. @@ -31,7 +31,7 @@ In service of building an end-to-end decentralized experience, dYdX has built th - **Mobile**: The iOS and Android apps are built in native Swift and Kotlin, respectively. The mobile apps interact with the Indexer in the same way the web application does, and will send trades directly to the chain. The mobile apps have been open sourced as well, allowing anyone to deploy the mobile app to the App Store or Play store. Specifically for the App store, the deployer needs to have a developer account as well as a Bitrise account to go through the app submission process. ### Lifecycle of an Order -Now that we have a better understanding of each of the components of dYdX Chain, let’s take a look at how it all comes together when placing an order. When an order is placed on v4, it follows the flow below: +Now that we have a better understanding of each of the components of dYdX Chain, let’s take a look at how it all comes together when placing an order. When an order is placed on dYdX Chain, it follows the flow below: 1. User places a trade on a decentralized front-end (e.g., website) or via API 2. The order is routed to a validator. That validator gossips that transaction to other validators and full nodes to update their orderbooks with the new order.