close tikv#13941, ref pingcap/tidb#39620

- If failed to get initial snapshot, remove the subscription as soon as possible. 
- Added a cache of getting checkpoint. This cache is lease-based -- the lease time is simply the tick interval of the coordinator. 
- Make the channel size huger for don't blocking the main loop when many regions migrating.

Signed-off-by: hillium <[email protected]>
Signed-off-by: hillium <[email protected]>
Signed-off-by: 山岚 <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…v#13984)

ref tikv#12842

Signed-off-by: tabokie <[email protected]>

ref tikv#12842

Signed-off-by: tabokie <[email protected]>
Signed-off-by: Xinye Tao <[email protected]>

ref tikv#12842

Implement periodical purge in v2.

Signed-off-by: tabokie <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

If operations like snapshot, split, are aborted by restart, they needs
to be either resumed or cleanup. This PR checks for garbage after
restart and resume committed operations.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>
Signed-off-by: Zwb <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

a few panic fix
1) update_approximate_raft_log_size may run into divid by zero error 
2) appy_delete may have None write_batch
3) StoreMeta::set_region may run into region corruption error if it's destroyed and re-created. 
4) TabletSnapManager's snapshot size calculation may throw Other error.

Signed-off-by: qi.xu <[email protected]>
Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: qi.xu <[email protected]>
Co-authored-by: Jay Lee <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>

…13995)

ref tikv#12842

Whenever timeout, the peer will check for unapplied logs whether
there are pending conf change and trigger heavy reads. So we
wait till most logs are applied before ticking.

It also fix following issues:
- PersistenceListener is not installed
- implementation of persisted_apply_index is wrong
- parse tablet name is wrong

Signed-off-by: Jay Lee <[email protected]>

ref tikv#12842

Signed-off-by: bufferflies <[email protected]>

close tikv#13997

Support to use evict_entry_cache when restart node.

Signed-off-by: tabokie <[email protected]>
Signed-off-by: hongyunyan <[email protected]>
Signed-off-by: Xinye Tao <[email protected]>
Signed-off-by: Jay Lee <[email protected]>
Signed-off-by: Wenbo Zhang <[email protected]>
Signed-off-by: Zwb <[email protected]>

Co-authored-by: Xinye Tao <[email protected]>
Co-authored-by: Jay <[email protected]>
Co-authored-by: Zwb <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

The API is supposed to be used with `append` but nowhere can we find
the clue. This PR merges `cut_logs` and `append` to reduce confusion
and mistakes.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

Publish tablet in apply thread is unsafe. This PR moves the operation to
raftstore. It also fixes the issues that applying two splits at a time can
cause panic. It also makes sure cache will be cleared after tablet is published.

Signed-off-by: Jay Lee <[email protected]>

ref tikv#12842

These two are helpers to utilize the static KV pairs in logger. In the
past, we use `logger.list()` to try to format the configured KV pairs,
but it will not work as values are omitted.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13730

Introduce priority-based channel

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

When the tablet contains dirty data right after split, generating snapshot may
just a waste. On the other hand, split usually happens on all peers, so delay
it a bit actually makes all peers more likely to be initialized by split. So
this PR rejects generating snapshot when it detects it still has dirty data.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

Signed-off-by: Jay Lee <[email protected]>

ref tikv#12842

1. store heartbeat should add snapshot and kv engine used size

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Xinye Tao <[email protected]>

close tikv#13977

Signed-off-by: tabokie <[email protected]>

ref tikv#12842

Make apply adaptive to reduce high tail latency.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

- add water metrics
- fix potential panic when destroying a peer
- fix incorrect store size

Signed-off-by: Jay Lee <[email protected]>

* util: Fix incorrect memory capacity

Signed-off-by: Wish <[email protected]>

* Fix lints

Signed-off-by: Wish <[email protected]>

* Check capacity with /proc/meminfo

Signed-off-by: Wish <[email protected]>

Signed-off-by: Wish <[email protected]>

* hotfix kvproto for global config

Signed-off-by: husharp <[email protected]>

* make format happy

Signed-off-by: husharp <[email protected]>

Signed-off-by: husharp <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13730

Signed-off-by: glorv <[email protected]>

…ock request (tikv#14037)

close tikv#14038, close pingcap/tidb#40114

Fixes the problem that when handling repeated acquire_pessimistic_lock requests is recevied, should_not_exist is ignored. 

TiKV provides idempotency for these RPC requests, but for acquire_pessimistic_lock, it ignored the possibility that the client may expect a pessimistic_rollback between two acquire_pessimistic_lock request on the same key. In this case the second request may come from another statement and carries `should_not_exist` that wasn't set in the previously finished pessimistic lock request. If the first request successfully acquired the lock and the pessimistic_rollback failed, TiKV may return a sucessful response, making the client believe that the key doesn't exist before. In some rare cases, this has risk to cause data inconsistency.

Signed-off-by: MyonKeminta <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

This PR fixes several bugs and metrics:
- Now waterfall timer will be reset in before_write, the goal is to solve
    the confusion that stall writes can pollute the whole waterfall metrics.
- Perf context is changed not to be associated with engine instance. Perf
      context is thread local and instance independent under the hook.
- Fix flushed index advance failure due to suspicious flush.
- Support print long uncommitted logs and fix incorrect commit time

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

Move transaction related code to txn_ext.rs.

Fix the bug that snapshot doesn't set term and extra_op.

Signed-off-by: Jay Lee <[email protected]>

ref tikv#12842

1) add snapshot apply metrics
2) disable bloomfilter for raftkv-v2 for now until a proper ratio is found 
3) disable rocksdb write stall for raftkv-v2 until the tablet flow control is fully verified.

Signed-off-by: Qi Xu <[email protected]>

Co-authored-by: Qi Xu <[email protected]>

ref tikv#12876

fix witness raft log gc panic and refactor

Signed-off-by: Wenbo Zhang <[email protected]>

Co-authored-by: Xinye Tao <[email protected]>

ref tikv#12999

copr: support handling keyspace request

Signed-off-by: iosmanthus <[email protected]>

ref tikv#13730

Support priority-based scheduling for the scheduler worker pool.

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: Xinye Tao <[email protected]>

ref tikv#11312

Fix `make docker`.

Signed-off-by: tabokie <[email protected]>

close tikv#14063

make sure kvproto compatibility

Signed-off-by: husharp <[email protected]>

ref tikv#12842

Signed-off-by: tabokie <[email protected]>

close tikv#14071

Log backup would aware TLS certifications changing.

Signed-off-by: hillium <[email protected]>

close tikv#14069

Signed-off-by: YangKeao <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

* support switch witness

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* add switch witness api for test_pd_client

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* pd heartbeat resp support switch witness

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* update region epoch

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* fix write apply state race

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* remove unnecessary code

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* add back test_witness_conf_change

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* add some tests

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* avoid test failures

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* a few refactor

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* add witness election priority and address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* clean code

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* fix tests failed caused by cfg

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* fix test failed caused by mistake modify

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* adjust priority after snapshot persisted

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* address comments

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* notify pd after switch witness as region changed

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* define a new backoff error for witness

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* fix panic caused by applygap

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* forbid transfer leader to non-witness waiting data

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* update kvproto

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* fix two panics

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* retry request snapshot

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* retry to request snaphost after term change

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

* update kvproto comment

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

Signed-off-by: Wenbo Zhang <[email protected]>
Signed-off-by: Zwb <[email protected]>
Co-authored-by: Xinye Tao <[email protected]>

close tikv#13983

- add etcd mock for pd
- add service for resource group

Signed-off-by: husharp <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13730

Support priority-based scheduling for the apply batch system.

Signed-off-by: Connor1996 <[email protected]>

close tikv#13586, close tikv#13752, ref tikv#14017

Signed-off-by: Andrei Dragus <[email protected]>

…#14087)

close tikv#14086

PD_REQUEST_HISTOGRAM_VEC can be changed to use static metrics to improve performance.

Signed-off-by: husharp <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13936

Signed-off-by: Zhi Qi <[email protected]>

…ikv#14088)

close tikv#14045

- Store the flashback `start_ts` in region meta.
- Allow the read request with a smaller ts during flashback.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…ata. (tikv#14046)

ref tikv#12999

import: sst_importer support download SST and rewrite into keyspace data.

Signed-off-by: iosmanthus <[email protected]>

close tikv#14052

The writecf and defaultcf's default block size is changed to 16KB to improve read performance (reduce read amplification)

Signed-off-by: qi.xu <[email protected]>

Co-authored-by: qi.xu <[email protected]>

ref tikv#12842

add an alias partitioned-raft-kv for RaftKv2

Signed-off-by: qi.xu <[email protected]>

Co-authored-by: qi.xu <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14092

resolved_ts: reduce network traffic by filter regions

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…#14101)

ref tikv#12842

If it accepts more logs than conf remove itself, applied_index == commit_index
will never be true. So we should check if it's a tombstone already first.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14109

Signed-off-by: you06 <[email protected]>

close tikv#14120

resource_control: unify wru/rru to ru

Signed-off-by: nolouch <[email protected]>

close tikv#14104

We need to use the new field to support item value as bytes to avoid proto string check failures.

Signed-off-by: husharp <[email protected]>

ref tikv#12842

- Update raft-engine to fix data corruption during restart
- Add tablet logger so we can know which tablet the logs belongs to

Signed-off-by: Jay Lee <[email protected]>

ref tikv#13855

Introduce observers when receive raft message.

Signed-off-by: CalvinNeo <[email protected]>

close tikv#14099

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842, ref tikv#13818

In V1, a peer is responsible to destroy itself. The design is to make leader do
less work and reduce writes. But from the practice of the pass years, not
making it a strong guarantee actually makes the implementation
complicated and hard to be correct and difficult to understand.

In V2, we changes to make leader the very role to make sure all removed
peers or merged peers must be destroyed in the end. Push mode is way
easier to understand and implement correctly. The downside is extra writes
are introduced but it's worthy.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13730

Support priority-based scheduling for the async write. Each channel of async write worker is replaced with a priority-based channel when the config `resource-control.enabled` is true.

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14093

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12876

Signed-off-by: Wenbo Zhang <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

set block-cache.capacity to 0.3 * memory limit instead of 0.45 for partitioned-raft-kv. 
set region-split-size to 1GB by default when bucket is enabled 
set region-split-size to 10GB by default when partitioned-raft-kv is enabled.

These numbers may be tuned futher if we have other better results.

Signed-off-by: qi.xu <[email protected]>

Co-authored-by: qi.xu <[email protected]>

ref tikv#13867, ref pingcap/tidb#38775

…#14140)

close tikv#14141

rawkv: fix flaky integration test case `test_raw_put_key_guard`.

Signed-off-by: Ping Yu <[email protected]>

ref tikv#12842

Add proc-macro test_case to reuse test cases.

Signed-off-by: SpadeA-Tang <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…k commit check failed (tikv#14145)

close tikv#14143, ref tikv#14143

As tikv#14143 mentioned, flashback should not
return `TxnLockNotFound` error to the client if the flashback commit check failed,
which will cause TiDB to retry the flashback forever. This PR changes this error to
`FlashbackNotPrepared` to match the client handling logic.

Signed-off-by: JmPotato <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…3965)

close tikv#13964

Support dynamically modify the count of async-ios.

Signed-off-by: Lucasliang <[email protected]>

close tikv#14115

Signed-off-by: tabokie <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

tikv#14077)

ref tikv#14151

Add a read pool time slice inspector to predict the waiting time for
read requests. Use the estimated duration to reject requests which
have busy_threashold.

Signed-off-by: Yilin Chen <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14128

1. release tablet in TableCache after peer destroyed
2. release tombstone tablet after applying snapshot

Signed-off-by: bufferflies <[email protected]>

Co-authored-by: Xinye Tao <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14142

apply: ignore compute and verify hash when it's a witness

Signed-off-by: Wenbo Zhang <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…s configuration (tikv#14153)

close tikv#14017

Signed-off-by: fengou1 <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13690

* Read pool size limit is removed. It was previously hard-coded to be
  the total number of cores. Limiting the number of threads to be less
  than the number of cores reduces the chance of context switch, but
  also makes TiKV prone to starvation problem (since read pool is
  currently FIFO), even if all the queries require no IO or IO is async
  (imagining there are number_of_cores clients sending big queries, and
  another numbers of clients sending small queries concurrently to a
  single TiKV server, and all of them require no IO). Thread starvation
  causes high tail latency which is even worse than context switching.
  According to the feature requester, increasing the number of threads
  significantly improved the tail latency in their environment. Thus, we
  should remove the limit, and leave it to the users.

Signed-off-by: Yang Zhang <[email protected]>

close tikv#14156, close pingcap/tidb#40341

copr: fix error when cast const Enum to any type

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14060

Support replica read for backup

Signed-off-by: Yang Zhang <[email protected]>

close tikv#14159

Signed-off-by: hillium <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14113

`OnFlushComplete` can be called out of order. What we can assume is
when a seqno finishes flush, all SSTs have smaller seqno must also
finish flush.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#14162, close tikv#14163

Force gc all split tablets by checking finish event.

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…the feature witness) (tikv#14155)

ref tikv#14137

Signed-off-by: joccau <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

Signed-off-by: tabokie <[email protected]>

…kv#14127)

close tikv#14165

Signed-off-by: hillium <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13889

Signed-off-by: hillium <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13730

Add grpc resource group QPS panel and fix datasource

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

…v#14170)

ref tikv#13730

Signed-off-by: glorv <[email protected]>

ref tikv#14131

PD schedulers: new scheduler `evict-slow-trend-scheduler`, for new slow store detecting and leader evicting

Signed-off-by: Liu Cong <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#14157

Signed-off-by: ekexium <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

Signed-off-by: tabokie <[email protected]>

…ikv#14192)

close tikv#14191

pass missing resource group name to request header

Signed-off-by: Connor1996 <[email protected]>

Co-authored-by: Xinye Tao <[email protected]>

…14018)

close tikv#14012

Added a "hibernate mode" for the log backup observer: while there isn't any task, it won't emit leader drop or region change events. So some verbose logs can be omitted.

Signed-off-by: hillium <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

close tikv#13926

Signed-off-by: Jay Lee <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#13936

Signed-off-by: Zhi Qi <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>

ref tikv#12842

materialize the region_split_size in optimize_for

Signed-off-by: qi.xu <[email protected]>

Co-authored-by: qi.xu <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>