[Epic] Secondary Indexes

[jsimnz]

Currently, only Primary Indexes are implemented, which is a simple relation between dockey/field => state_data. Effectively this is an index on the dockey field of a document. This is a minimal requirement for making the DB work, however we need to let devs also manually define "Secondary Indexes".

Secondary Indexes allow devs to index on alternative fields, even a combination of multiple fields. Which can lead to effecient querys for certain filter/order/etc arguments. The idea is if a developer knows ahead of time the likely kind of queries/lookups their respective application needs, then they can optimize the DB structure for those specific queries.

Starting a spec doc as there are a number of goals/compromises/designs that need to be considered: https://hackmd.io/rLHMu-KvRFC0OoE8xkwYlA

Tasks

Preview Give feedback

1. Evaluate probable performance differences between secondary index key structure #231
   area/db-system perf question +3
   
2. Sec. Indexes: Implement non-unique indexes (querying excluded) #297
   area/query feature priority/high +3
   
3. Sec. Indexes: CLI commands for creating and dropping indexes #1553
   area/api area/cli feature priority/high +4
   
4. Sec. Indexes: Make queries use indexes for fetching data #1555
   area/query feature perf priority/high +4
   
5. Sec. Indexes: Implement unique indexes #298
   area/query feature priority/high +3
   
6. Sec. Indexes: Implement composite indexes #299
   area/query feature priority/high +3
   
7. Sec. Indexes: Allow backfill #296
   area/query feature +2
8. Cleanup unused stuff from Descriptions/fetcher #307
   refactor +1
   
9. Sec. Indexes: implement caching of indexes for collections #1585
   area/collections perf +2
10. Sec. indexes: enable multi-indexed queries #2132
    area/query feature perf +3
    
11. Sec. indexes: enable order direction for composite indexes #2229
    area/datastore area/query feature perf +4
    
12. Sec. indexes: Allow multiple fields with nil values with unique index #2180
    area/datastore feature +2
    
13. Panic if given an index with a name that already exists #2247
    bug +1
14. Violating a unique index with a nil value returns unhelpful error, or panics #2174
    area/db-system bug code quality +3
    
15. Sec. Index: enable indexing of elements of array fields #2279
    area/collections feature perf +3
    
16. Sec. Index: enable indexing of key-values of JSON fields #2280
    area/collections feature perf +3
    
17. Sec. index: enable indexing of related objects #2601
    area/query perf +2
    
18. Sec. index: analyse and choose most optimal sec. index for fetching #2680
    area/query perf +2
19. Sec. index: make compound filters utillize sec. indexes #3299
    area/query feature perf +3
20. Sec. index: copy scan filter conditions only arrays #3306
    area/query perf +2
21. Sec. index: make json array elements be encoded with depth information #3329
    area/datastore area/query perf +3

[AndrewSisley]

Had a thought - if/when implementing the ds.Get call chain, do it in a feature-agnostic way that allows fetcher to use ds.Get when it can