Tommy garch patch 1

pages/Architecture/_meta.json

@@ -0,0 +1,4 @@
+{
+    "architectural_overview": "Architectural Overview",
+    "indexer": "Indexer"
+  }

pages/Architecture/architectural_overview.md

@@ -0,0 +1,46 @@
+# Intro to dYdX v4 Architecture
+
+v4 (or dYdX Chain) is the next iteration of the dYdX protocol, which will consist of open-source software. The current version live in production today is known as v3 and can be found [here](https://trade.dydx.exchange/?utm_source=dydx-website&utm_medium=blog&utm_content=v4-technical-architecture-overview). v3 and past versions of dYdX were, at their core, smart contracts deployed to existing chains, combined with centralized services hosted in the cloud. v4 will be a standalone L1 blockchain, featuring a fully decentralized, off-chain orderbook and matching engine. The dYdX Chain will be based on the Cosmos SDK and CometBFT PoS consensus protocol.
+
+As we approach the release of v4 mainnet open-source software (“dYdX Chain”), we want to provide a peek into what the dYdX team is building. This post presents a high-level overview of the architecture of v4. Given that v4 is still under development, please note that what is presented below is subject to change.
+
+### v4 System Architecture
+dYdX v4 is being designed to be completely decentralized end-to-end. The main components broadly include the protocol, the Indexer, and the front end. Each of these components will be made available as open source software. None of the components will be run by dYdX Trading Inc.
+
+![image](https://github.com/dydxprotocol/v4-documentation/assets/130097657/e9a54253-e7fa-44ab-97c5-ae1ce7cae320)
+
+### Protocol (or “Application”)
+The open-source protocol is an L1 blockchain built on top of [CometBFT](https://dydx.exchange/blog/v4-technical-architecture-overview#:~:text=on%20top%20of-,CometBFT,-and%20using%20CosmosSDK) and using [CosmosSDK](https://v1.cosmos.network/sdk). The node software is written in Go, and compiles to a single binary. Like all CosmosSDK blockchains, v4 uses a proof-of-stake consensus mechanism. 
+
+The protocol will be supported by a network of nodes. There are two types of nodes:
+
+- **Validators**: Validators are responsible for storing orders in an in-memory orderbook (i.e. off chain and not committed to consensus), gossipping transactions to other validators, and producing new blocks for the dYdX Chain through the consensus process. The consensus process will have validators take turns as the proposer of new blocks in a weighted-round-robin fashion (weighted by the number of tokens staked to their node). The proposer is responsible for proposing the contents of the next block. When an order gets matched, the proposer adds it to their proposed block and initiates a consensus round. If ⅔ or more of the validators (by stake weight) approve of a block, then the block is considered committed and added to the blockchain. Users will submit transactions directly to validators.
+
+- **Full Nodes**: A Full Node represents a process running the v4 open-source application that does not participate in consensus. It is a node with 0 stake weight and it does not submit proposals or vote on them. However, full nodes are connected to the network of validators, participate in the gossiping of transactions, and also process each new committed block. Full nodes have a complete view of the dYdX Chain and its history, and are intended to support the Indexer. Some parties may decide (either for performance or cost reasons) to run their own full node and/or Indexer.
+
+### Indexer
+The Indexer is a read-only collection of services whose purpose is to index and serve blockchain data to end users in a more efficient and web2-friendly way. This is done by consuming real time data from a v4 full node, storing it in a database, and serving that data through a websocket and REST requests to end-users.
+
+While the v4 open-source protocol itself is capable of exposing endpoints to service queries about some basic on-chain data, those queries tend to be slow as validators and full nodes are not optimized to efficiently handle them. Additionally, an excess of queries to a validator can impair its ability to participate in consensus. For this reason, many Cosmos validators tend to disable these APIs in production. This is why it is important to build and maintain Indexer and full-node software separate from validator software.
+
+Indexers will use Postgres databases to store on-chain data, Redis for off-chain data, and Kafka for consuming and streaming on/off-chain data to the various Indexer services.
+
+### Front-ends
+In service of building an end-to-end decentralized experience, dYdX is building three open-source front ends: a web app, an iOS app, and an Android app.  
+
+- **Web application**: The website will be built using Javascript and React. The website will interact with the Indexer through an API to get off-chain orderbook information and will send trades directly to the chain. dYdX will open source the front end codebase and associated deployment scripts. This will allow for anyone to easily deploy and access the dYdX front end to/from their own domain/hosting solution via IPFS/Cloudflare gateway. 
+
+- **Mobile**: The iOS and Android apps are built in native Swift and Kotlin, respectively. The mobile apps will interact with the Indexer in the same way the web application does, and will send trades directly to the chain. The mobile apps will be open sourced as well, allowing anyone to deploy the mobile app to the App Store or Play store. Specifically for the App store, the deployer will need to have a developer account as well as a Bitrise account to go through the app submission process.
+
+### Lifecycle of an Order
+Now that we have a better understanding of each of the components of dYdX v4, let’s take a look at how it all comes together when placing an order. When an order is placed on v4, it will follow the flow below:
+
+1. User places a trade on a decentralized front-end (e.g., website) or via API
+2. The order is routed to a validator. That validator gossips that transaction to other validators and full nodes to update their orderbooks with the new order.
+3. The consensus process picks one validator to be the proposer. The selected validator matches the order and adds it to its next proposed block.
+4. The proposed block continues through the consensus process. 
+    1. If ⅔ of validator nodes vote to confirm the block, then the block is committed and saved to the on-chain databases of all validators and full nodes.
+    2. If the proposed block does not successfully hit the ⅔ threshold, then the block is rejected.
+5. After the block is committed, the updated on-chain (and off-chain) data is streamed from full nodes to Indexers. The Indexer then makes this data available via API and Websockets back to the front end and/or any other outside services querying for this data.
+
+The flow above is a high level overview of how an order/data will move through v4. We will do further deep dives into the protocol, indexer, and various front ends’ infrastructure in subsequent blog posts as we get closer to the release of v4 mainnet open-source software.

pages/Architecture/indexer.md

@@ -0,0 +1,102 @@
+# Indexer Deep Dive
+
+A good way to think about the indexer is as similar to Infura or Alchemy’s role in the Ethereum ecosystem. However, unlike Infura/Alchemy, and like everything else in dYdX v4, the Indexer is completely open source and can be run by anyone!
+
+### What is the Indexer?
+
+As part of tooling for the dYdX ecosystem, we want to ensure that clients have access to performant data queries when using the dYdX v4 exchange. Cosmos SDK Full Nodes offer a number of APIs that can be used to request on-chain data. However, these Full Nodes are optimized for committing and executing blocks, not for serving high frequency, low-latency requests from web/mobile clients.
+
+This is why we wrote software for an indexing service. The Indexer is a read-only service that serves off chain data to clients over REST APIs and Websockets. Its purpose is to store and serve data that exists on the dYdX Chain in an easier to use way. In other words, the purpose of an indexer is to index and serve data to clients in a more performant, efficient and web2-friendly way. For example the indexer will serve websockets that provide updates on the state of the orderbook and fills. These clients will include front-end applications (mobile and web), market makers, institutions, and any other parties looking to query dYdX Chain data via a traditional web2 API.
+
+### On-chain vs. Off-chain data
+
+The Indexer will run two separate ingestion/storage processes with data from a v4 Full Node: one for on-chain data and one for off-chain data. Currently, throughput of on-chain data state changes is expected to be from 10-50 events/second. On the other hand, the expected throughput of off-chain data state changes is between 500-1,000 events/second. This represents a 10-100x difference in throughput requirements. By handling these data types separately, v4 is built to allow for different services to better scale according to throughput requirements.
+
+### On-chain Data
+
+On-chain data is all data that can be reproduced by reading committed transactions on the dYdX Chain. All on-chain data has been validated through consensus. This data includes:
+
+1. Account balances (USDC)
+2. Account positions (open interest)
+3. Order Fills 
+    1. Trades 
+    2. Liquidations
+    3. Deleveraging
+    4. Partially and completely filled orders
+4. Funding rate payments
+5. Trade fees
+6. Historical oracle prices (spot prices used to compute funding and process liquidations)
+7. Long-term order placement and cancellation
+8. Conditional order placement and cancellation
+
+### Off-chain Data
+
+Off-chain data is data that is kept in-memory on each v4 node. It is not written to the blockchain or stored in the application state. This data cannot be queried via the gRPC API on v4 nodes, nor can it be derived from data stored in blocks. It is effectively ephemeral data on the v4 node that gets lost on restarts/purging of data from in-memory data stores. This includes:
+
+1. Short-term order placement and cancellations
+2. Order book of each perpetual exchange pair
+3. Indexed order updates before they hit the chain
+
+## Indexer Architecture
+
+![image](https://github.com/dydxprotocol/v4-documentation/assets/130097657/8fc9842f-49e7-430f-a1f0-969c72489b28)
+
+The Indexer is made up of a series of services that ingest information from v4 Full Nodes and serve that information to various clients. Kafka topics are used to pass events/data around to the services within the Indexer. The key services that make up Indexer are outlined below.
+
+### Ender (On-chain ingestion)
+
+Ender is the Indexer’s on-chain data ingestion service. It consumes data from the “to-ender” Kafka topic (which queues all on-chain events by block) and each payload will include all event data for an entire block. Ender takes all state changes from that block and applies them to a Postgres database for the Indexer storing all on-chain data. Ender will also create and send websocket events via a “to-websocket-?” Kafka topic for any websocket events that need to be emitted.
+
+### Vulcan (Off-chain ingestion)
+
+Vulcan is the Indexer’s off-chain data ingestion service. It will consume data from the “to-vulcan” Kafka topic (queues all off-chain events), which will carry payloads that include active order book updates, place order updates, cancel order updates, and optimistic fills. This data will be stored in a Redis cache. Vulcan will update Redis with any new open orders, set the status of canceled orders to cancel pending, and update orderbooks based on the payload received. Vulcan will also update Postgres whenever a partially filled order is canceled to update the state of the order in Postgres. Vulcan will also create and send websocket events via a “to-websocket-?” Kafka topic for any websocket events that need to be emitted.
+
+### Comlink (API Server)
+
+Comlink is an API server that will expose REST API endpoints to read both on-chain and off-chain data. For example, a user could request their USDC balance or the size of a particular position through Comlink, and would receive a formatted JSON response.
+
+As an explicit goal set out by the dYdX team, we’re designing v4 APIs to closely match the [v3 exchange APIs](https://dydx.exchange/blog/v4-deep-dive-indexer#:~:text=closely%20match%20the-,v3%20exchange%20APIs,-.%20We%20have%20had). We have had time to gather feedback and iterate on these APIs over time with v3, and have confidence that they are reasonable at the product-level.
+
+### Roundtable
+
+Roundtable is a periodic job service that provides required exchange aggregation computations. Examples of these computations include: 24h volume per market, open interest, PnL by account, candles, etc. 
+
+### Socks (Websocket service)
+
+Socks is the Indexer’s websockets service that allows for real-time communication between clients and the Indexer. It will consume data from ender, vulcan, and roundtable, and send websocket messages to connected clients.
+
+## Hosting & Deploying the Indexer
+
+In service of creating an end-to-end decentralized product, the Indexer will be open source.  This will include comprehensive documentation about all services and systems, as well as infrastructure-as-code for running the Indexer on popular cloud providers. 
+
+The specific responsibilities of a third party operator looking to host the Indexer generally include initial deployment and ongoing maintenance. 
+
+Initial deployment will involve: 
+
+- Setting up AWS infrastructure to utilize the open-source repo.
+- Deploying Indexer code to ingest data from a full-node and expost that information through APIs and websockets
+- Datadog (provides useful metrics and monitoring for Indexer services), and Bugsnag (real-time alerting on bugs or issues requiring human intervention).
+
+Maintenance of the Indexer will involve:
+
+- Migrating and/or upgrading the Indexer for new open-source releases
+- Monitoring Bugsnag and Datadog for any issues and alerting internal team to address
+- Debugging and fixing any issues with a run book provided by dYdX
+
+dYdX believes that, at minimum, a DevOps engineer will be required to perform the necessary duties for deployment and maintenance of the Indexer. An operator will need to utilize the services below: 
+
+- AWS
+    - ECS - Fargate
+    - RDS - Postgres Database
+    - EC2
+    - Lambda
+    - ElastiCache Redis
+    - EC2 ELB - Loadbalancer
+    - Cloudwatch - Logs
+    - Secret Manager
+- Terraform Cloud - for deploying to the cloud
+- Bugsnag - bug awareness
+- Datadog - metrics and monitoring
+- Pagerduty - alerting
+
+Operators should be able to host the open-sourced Indexer for public access in a highly available (i.e., high uptime) manner. Requirements include owning accounts to the services above and hiring the appropriate personnel to perform deployment and maintenance responsibilities. 

pages/FAQ.md

@@ -0,0 +1,7 @@
+# FAQ Documentation
+
+Since the inception of v4, we have been compiling a list of relevant questions and answers that are asked by MMs/Trading Firms on Telegram. We turned this list into a live _v4 Telegram Questions_ document, which should be able to answer many of your more nuanced questions.
+
+It's a large document, so we recommend CTRL+F to find key words. 
+
+You can anonymously access the live document with [this link](https://docs.google.com/document/d/1QZE5aemQ1lzBN_2RA8-tTVk5JVQ_fNNB4VeSEWKWl8w/edit?usp=sharing).

pages/_meta.json

@@ -1,6 +1,7 @@
 {
   "index": "Introduction",
   "about": "About dYdX",
+  "architecture": "Architecture"
   "validators": "Validators",
   "network": "Network Configuration",
   "trading": "Trading",
@@ -11,6 +12,7 @@
   "governance": "Governance",
   "security": "Security",
   "guides":"Guides",
+  "FAQ": "FAQs",
   "terms_and_policies": "Terms and Policies",
   "app": {
     "title": "Testnet ↗",

pages/guides/_meta.json

@@ -1,6 +1,9 @@
 {
     "how_to_set_up_full_node": "How to set up a full node",
     "how_to_uncross_orderbook": "How to uncross the orderbook",
-    "how_to_send_usdc_to_dydx": "How to send USDC from Ethereum to dYdX"
+    "how_to_send_usdc_to_dydx": "How to send USDC from Ethereum to dYdX",
+    "how_to_interpret_block_data_for_trades": "How to interpret block data for trades",
+    "maintaining_a_local_orderbook": "Maintaining a Local Orderbook",
+    "using_Cosmovisor_to_stage_dYdXChain": "Using Cosmovisor to stage dYdX Chain binary" 
   }
-
+

pages/guides/how_to_interpret_block_data_for_trades.md

@@ -0,0 +1,53 @@
+## dYdX v4: How to Interpret the Block Data for Trades
+
+<img width="639" alt="Screenshot 2024-03-29 at 2 34 06 PM" src="https://github.com/dydxprotocol/v4-documentation/assets/130097657/19bc0173-0d79-473f-b88a-f99b72c7065a">
+
+In dYdX trading, quantities and prices are represented in quantums (for quantities) and subticks (for prices), which need conversion for practical understanding.
+
+### Quantums
+
+The smallest increment of position size. Determined from `atomicResolution`.
+
+atomicResolution - Determines the size of a quantum. [For example](https://github.com/dydxprotocol/v4/blob/08069ba905753158b9f390ca52e3f9f0fb2cb3d5/config.yml#L101), an `atomicResolution` of 10 for `BTC`, means that 1 quantum is `1e-10` `BTC`.
+
+### Subticks
+
+Human-readable units: `USDC/<currency>` e.g. USDC/BTC
+
+Units in V4 protocol: `quote quantums/base quantums` e.g. (`1e-14 USDC/1e-10 BTC`)
+
+Determined by `quantum_conversion_exponent`, this allows for flexibility in the case that an asset’s prices plummet, since prices are represent in subticks, decreasing `subticks_per_tick` would allow for ticks to denote smaller increments between prices.
+
+E.g. 1 `subtick` = `1e-14 USDC/1e-10 BTC`  and if BTC was at 20,000 USDC/BTC, a `tick` being 100 USDC/BTC (`subtick_per_tick` = 10000) may make sense.
+
+If BTC drops to 200 USDC/BTC, a `tick` being 100 USDC/BTC no longer makes sense, and we may want a `tick` to be 1 USDC/BTC, which lets us set `subtick_per_tick` to 100 to get to a `tick` size of 1 USDC/BTC.
+
+### Now back to the interpretation of the above image:
+
+1. First, notice column I is negative.  That means this trade is a sell by the taker account.  If It was positive, it would be a buy.
+
+Result: Determined if this is a buy or a sell
+
+2. Next, look at column N.  The perpetual_id is 7, which maps to AVAX-USD market.  You can see all the mappings from this endpoint https://indexer.dydx.trade/v4/perpetualMarkets where the clobPairId is the perpetual_id.
+
+Result: Determined the market
+
+3. Next, we need to get the decimals for this market.  First, get the atomicResolution from that endpoint above which we see is -7.  Now we can get the size of the trade.  From column I and J, take this number -500000000 and multiply by 10^(AtomicResolution) and you get: -500000000 x 10^-7 = 50, so the quantity is 50.
+
+Result: Determined the quantity
+
+4. Next, look at columns, E, F, G, H, I, and J
+
+<img width="619" alt="Screenshot 2024-03-29 at 2 37 28 PM" src="https://github.com/dydxprotocol/v4-documentation/assets/130097657/44661861-3ec0-40fb-951d-43c3ce9bb015">
+
+The price of the trade is either abs((G+E)/I)*10e(-6 - AtomicResolution), or abs((H+F)/J)*10e(-6 - AtomicResolution), either one is the same.  Note that the ‘-6’ is because the AtomicResolution of USDC is -6.
+
+abs((1479130125 + 369875)/-500000000)*10e(-6 + 7) = 29.59
+
+abs((-1479337255 - 162745)/500000000)*10e(-6 +7) = 29.59
+
+Result: Determined the price
+
+### Conclusion
+
+In conclusion, we have determined that this trade is SELL 50 AVAX-USD at price $29.59