First release objectives

[v3g42]

First release Objectives

1. dozer cli experience

// Calls gRPC methods on management process
dozer api start | stop | stats | generate-auth | logs (tail, 100 lines)
dozer app start | stop | restart | stats | logs(tail, 100 lines)  --name=eth
dozer ps


// Cli commands
-h help
-c configuration
-v version

• Separate cache reader and writer into orchestrator and Api process.
• Include base log4rs setup within the binary.
• Split logs in meaningful folder structure.
• Graceful shutdown of pipelines
• dozer ps should show messages processed, running time and basic stats.
• Implement checkpointing.
• Take api configuration in config file including port, cors, auth etc
• Orchestrator

2. SQL Support

a) SELECT
b) GROUP BY

• SUM
• COUNT
• MIN
• MAX
• AVG
  c) WHERE
  d) HAVING
  e) JOIN
  f) DISTINCT
  f) Expressions (https://www.postgresql.org/docs/9.2/functions.html)

3. Cache

• Dynamic index creation
• Use Multiple indexes to return results.
• Cache Query Streaming
• Auth scenarios and tests

4. REST & gRPC

• Streaming filter using query expression
• Auth Support for gRPC

5. Tracing

• Instrument everything
• Store in cache
• Support Open Telemetry
• CLI should ask for permission

6. Ingestion

• Remove Rocksdb and global counter
• Individual long running test cases for every connector in Nightly schedule.
• Validation of connection strings and variables (escape everything)
• Type conversions tests for all data types battle tested.
• Postgres: Minimum and Max version supoprt documented.
• Postgres: Run with scenarios when SSH is enabled.
• Postgres: AWS RDS connection
• Postgres: Connect to a different docker container that doesnt use debezium image.
• Postgres: Experience when replication slot is not available.
• Postgres: Debezium and Kafka variations.
• Snowflake: ODBC support across ubuntu, AWS linux etc (documentation on how to solve this properly)

7. Testing

• Performance benchmarks for every module (cache, apis, pipeline, e2e)
• Penetration testing for admin, APIs, gRPC.
• static code analysis for security vulneraibilities
• Document 100-200 sql scenarios that need to execute and compare against https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki
• Document 100-200 cache queries and compare against mongodb.

8. Samples

• Ubuntu and Mac User guides.
• Samples demonstrating Postgres / Snowflake / Ethereum connectors
• Samples demonstrating transformations
• Samples demonstrating auth configuration

9. Documentation

• Document all major entities (Api Endpoint, Application, Source, Connection)
• Pick a markdown tool

10. Admin UI

• Add Source workflow
• Add Pipeline workflow
• Dashboard