Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] main from dolthub:main #54

Merged
merged 110 commits into from
Feb 8, 2025
Merged
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
3c1a365
go/store/nbs: store.go: Add a mechanism to bracket outstanding read r…
reltuk Jan 15, 2025
f099fdc
go/store/nbs: chunkReader,chunkSource: GC: Add the ability to take de…
reltuk Jan 15, 2025
26c909d
go/store/nbs: Add GC keeper calls on reads through NomsBlockStore.
reltuk Jan 16, 2025
4a59dcb
go/store/nbs: tablePersister: Add GC dependency capturing to written …
reltuk Jan 16, 2025
78ee20a
go/store/nbs: table_set: append: Thread GC dependency tracking throug…
reltuk Jan 16, 2025
e423a5c
go/store/nbs: store.go: MarkAndSweepChunks: After adding read trackin…
reltuk Jan 16, 2025
6ad9b37
repofmt.sh.
reltuk Jan 16, 2025
5355928
Avoid loading DB for commands where it's not necessary.
nicktobey Jan 22, 2025
1d4a82f
Make dEnv.DoltDB private and add an accessor function.
nicktobey Jan 23, 2025
8dfadf9
Propagate context.
nicktobey Jan 23, 2025
9c04d50
Remove explicit list of commands that not need to eagerly load the DB.
nicktobey Jan 23, 2025
a963a85
Don't load the database until we actually bind the queryist.
nicktobey Jan 23, 2025
b6c233d
Remove explicit calls to `dEnv.ReloadDB()` from commands.
nicktobey Jan 23, 2025
e89385d
Add url parameter to LoadWithoutDB and remove LoadWithDeferredDB
nicktobey Jan 23, 2025
d43ce7f
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 23, 2025
849e5c0
Create context object for remote server main function.
nicktobey Jan 23, 2025
e568a66
Fix infinite recursion loop in DoltEnv.DoltDB()
nicktobey Jan 23, 2025
5c04d5f
go/store/nbs: store.go: Fix errgroup context usage-after-Wait bug in …
reltuk Jan 23, 2025
d0fb77f
Fixup tests.
nicktobey Jan 23, 2025
7f0ca8c
Respond to PR feedback.
nicktobey Jan 23, 2025
411b676
Respond to PR feedback.
nicktobey Jan 24, 2025
e540aff
Add context variable to tests.
nicktobey Jan 24, 2025
f78cc0d
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 24, 2025
8b10e7f
Update bats tests to assert that you can't clone directly from anothe…
nicktobey Jan 25, 2025
b830fbc
[ga-bump-dep] Bump dependency in Dolt by jycor (#8790)
coffeegoddd Jan 27, 2025
2d3105e
Add the dirty column to the dolt_branches system table
macneale4 Jan 27, 2025
809be27
Tests for the dolt_branches dirty column
macneale4 Jan 27, 2025
9da62eb
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
macneale4 Jan 27, 2025
fdc79a5
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 27, 2025
9b9722f
Avoid race conditions when loading dolt DB.
nicktobey Jan 27, 2025
5bf5642
Fix workbench tests to expect dirty column
macneale4 Jan 27, 2025
da7bb5c
Don't error out when no workingset is found
macneale4 Jan 27, 2025
fad9c1d
Merge pull request #8793 from dolthub/macneale4/dirty-branch-column
macneale4 Jan 27, 2025
bde0bf9
[ga-bump-release] Update Dolt version to 1.48.0 and release v1.48.0
coffeegoddd Jan 28, 2025
003e2d1
go: binlogreplication: Add Session{{Begin,End}Command,End} lifecycle …
reltuk Jan 28, 2025
89eb2af
go: cmd/dolt: sqlengine: Actually call DoltBinlogReplicaController.Cl…
reltuk Jan 28, 2025
4895747
[ga-bump-dep] Bump dependency in Dolt by zachmu
zachmu Jan 28, 2025
78e9a8a
Merge pull request #8760 from dolthub/aaron/gc-read-dependencies
reltuk Jan 28, 2025
1b59420
go/store/nbs: generational_chunk_store.go: In GCMode_Full, also take …
reltuk Jan 28, 2025
5b55d32
go: sqlserver: Clean up how remotesrv gets a handle to the mrEnv.File…
reltuk Jan 28, 2025
a6b1a26
go: sqle,remotesrv: Implement sql.Session lifecycle callbacks for sql…
reltuk Jan 28, 2025
c401029
Merge pull request #8794 from dolthub/aaron/sql-replication-session-l…
reltuk Jan 28, 2025
366e466
go: sqle/dprocedures: dolt_gc: Move to an intantiated instance.
reltuk Jan 21, 2025
915e392
go: sqle: dsess: Make DoltSession Lifecycle aware. Move towards a GCS…
reltuk Jan 22, 2025
ef954d0
go: sqle/dprocedures: dolt_gc: Implement a session aware safepoint co…
reltuk Jan 23, 2025
bdc8ff1
dolt_gc,dsess: Add VisitGCRoots to dsess.Session and call it from dol…
reltuk Jan 24, 2025
fc3217e
go: sqle: dolt_gc: Add DOLT_GC_SAFEPOINT_CONTROLLER_CHOICE env variab…
reltuk Jan 28, 2025
13c9ddf
go: sqlserver,binlogreplication: Clean up session usage a little to h…
reltuk Jan 29, 2025
caaa4bd
Merge pull request #8795 from dolthub/zachmu-3f5bb8c5
Hydrocharged Jan 29, 2025
819136d
go: dtables: help_table: Manually specify that dolt_gc exists for now.
reltuk Jan 29, 2025
d27b998
prevent nil panic on DoltEnv::loadDBOnce
nicktobey Jan 29, 2025
cda466f
/go/libraries/doltcore/env/actions: make iter resolved tags paginated…
coffeegoddd Jan 21, 2025
09dd814
/go/libraries/doltcore/env/actions/tag.go: reinstage iterresolved tag…
coffeegoddd Jan 30, 2025
8356ba7
go: store/types: Fix dolt_gc on databases that use vector indexes.
reltuk Jan 30, 2025
3dd50f5
Merge pull request #8796 from dolthub/aaron/gc-read-dependencies-oldg…
reltuk Jan 30, 2025
ddaa922
Fix typo in error message.
reltuk Jan 30, 2025
337671a
Merge pull request #8774 from dolthub/db/tags-3
coffeegoddd Jan 30, 2025
153d46b
go: sqle: gc_safepoint_controller: PR feedback, comments to explain s…
reltuk Jan 30, 2025
5826451
Merge remote-tracking branch 'origin/main' into aaron/dsess-lifecycle
reltuk Jan 30, 2025
7c1cff7
integration-tests/go-sql-server-driver: Fix bug in concurrent_gc_test.
reltuk Jan 30, 2025
f79db4a
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
reltuk Jan 30, 2025
b9f6a56
go: sqle: dprocedures: dolt_gc: Get the safepoint controller from the…
reltuk Jan 30, 2025
b1aa361
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 30, 2025
9fd5d56
[ga-bump-dep] Bump dependency in Dolt by Hydrocharged
Hydrocharged Jan 31, 2025
4a2606e
Merge pull request #8803 from dolthub/Hydrocharged-67aa2a43
Hydrocharged Jan 31, 2025
3821a7a
Include deleteErr message in returned error.
fulghum Jan 31, 2025
84bd03d
Merge pull request #8804 from dolthub/fulghum/delete-err
fulghum Jan 31, 2025
16ae96b
[no-release-notes] go: binlogreplication: tests: Parallelize the test…
reltuk Jan 31, 2025
f33c9fc
Merge pull request #8801 from dolthub/aaron/ivff-gc
reltuk Jan 31, 2025
0ef447e
Merge pull request #8797 from dolthub/aaron/sql-server-remotesapi-ses…
reltuk Jan 31, 2025
f131fb8
go/libraries/doltcore/sqle/binlogreplication: Fix windows build tags.
reltuk Jan 31, 2025
ed1af8f
Merge pull request #8805 from dolthub/aaron/binlog-tests-parallel
reltuk Jan 31, 2025
52b625f
Merge remote-tracking branch 'origin/main' into aaron/dsess-lifecycle
reltuk Feb 3, 2025
b44469d
go: sqle/remotesrv.go: Fix session lifecycle callbacks interceptors.
reltuk Feb 3, 2025
4e6378f
Merge pull request #8798 from dolthub/aaron/dsess-lifecycle
reltuk Feb 3, 2025
3d5d419
Update tag_test.go
nicktobey Feb 3, 2025
2911085
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Feb 3, 2025
3cf7df2
print start server log
liuliu-dev Feb 3, 2025
9dd401c
Deprecate user and pass options for sql-server
fulghum Jan 30, 2025
00cacf4
printf
liuliu-dev Feb 3, 2025
d1e90c4
Merge pull request #8806 from dolthub/liuliu/replace-cli-printErrf-wi…
liuliu-dev Feb 3, 2025
b4b6d71
Merge pull request #8783 from dolthub/nicktobey/lazy-load
nicktobey Feb 3, 2025
6d95550
go: sqle/binlogreplication: Add some waits for more reliable observat…
reltuk Feb 4, 2025
55001ef
Tidying up journal writer code comments and error handling
fulghum Jan 10, 2025
c0c9c64
PR Feedback: Making processJournalRecords() less aggressive on error'…
fulghum Jan 29, 2025
e701e5e
[no-release-notes] go: nbs/store: Small cleanups to iterateAllChunks.
reltuk Feb 4, 2025
56b42c9
[no-release-notes] go: store/nbs: GhostBlockStore: Add refCheck which…
reltuk Feb 4, 2025
04498d5
Merge pull request #8732 from dolthub/fulghum/journal
fulghum Feb 4, 2025
13b81b9
[no-release-notes] go: store/nbs: Unify logic from UpdateManifest and…
reltuk Feb 4, 2025
094fc7b
[kvexec] fix left join nullability bug (#8812)
max-hoffman Feb 4, 2025
3fa6a29
[kvexec] literal types cast to lookup expr type (#8808)
max-hoffman Feb 4, 2025
928ae32
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
reltuk Feb 4, 2025
a279a81
go: store/nbs: ghost_store.go: Fix bug in refCheck so it actually upd…
reltuk Feb 5, 2025
4398e55
Bug fix for encoding extended types in keys
zachmu Feb 2, 2025
9a2e242
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
zachmu Feb 5, 2025
e81bed9
[ga-bump-dep] Bump dependency in Dolt by max-hoffman (#8814)
coffeegoddd Feb 5, 2025
d766206
Merge pull request #8800 from dolthub/fulghum/rm-sql-server-user
fulghum Feb 5, 2025
5dece74
Merge pull request #8809 from dolthub/aaron/binlog-replica-test-sleeps
reltuk Feb 5, 2025
568590e
Merge pull request #8811 from dolthub/aaron/nbs-ghost-store-refcheck
reltuk Feb 5, 2025
8cc323e
[ga-bump-release] Update Dolt version to 1.49.0 and release v1.49.0
coffeegoddd Feb 5, 2025
cb155c5
Merge pull request #8810 from dolthub/aaron/nbs-iterate-all-chunks-stats
reltuk Feb 5, 2025
973ca47
Merge pull request #8813 from dolthub/aaron/nbs-manifest-update-dedup
reltuk Feb 5, 2025
cc43725
fix build break (#8819)
macneale4 Feb 5, 2025
9e36f51
Merge pull request #8817 from dolthub/zachmu/extended-encoding
zachmu Feb 5, 2025
edfa9c7
Expand itemWidth field to 32 bits.
nicktobey Feb 7, 2025
34f9158
Revert PR #8723 which disabled the smart JSON chunker when the docume…
nicktobey Feb 7, 2025
c6d7f65
Re-enable disabled json test.
nicktobey Feb 7, 2025
f703df9
Merge pull request #8831 from dolthub/nicktobey/jsonread
nicktobey Feb 7, 2025
5b028a5
[no-release-notes] fix bad tests and bump (#8837)
jycor Feb 7, 2025
13d1c2d
[ga-bump-release] Update Dolt version to 1.49.1 and release v1.49.1
coffeegoddd Feb 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
go/store/nbs: store.go: MarkAndSweepChunks: After adding read trackin…
…g, make it so that reading chunks as part of the GC process does not take further dependencies on them and never blocks on waitForGC.
  • Loading branch information
reltuk committed Jan 16, 2025
commit e423a5c28fd6b360ac040d1acdc93033ed1a448d
14 changes: 9 additions & 5 deletions go/store/nbs/generational_chunk_store.go
Original file line number Diff line number Diff line change
@@ -145,14 +145,18 @@ func (gcs *GenerationalNBS) GetMany(ctx context.Context, hashes hash.HashSet, fo
}

func (gcs *GenerationalNBS) GetManyCompressed(ctx context.Context, hashes hash.HashSet, found func(context.Context, CompressedChunk)) error {
return gcs.getManyCompressed(ctx, hashes, found, gcDependencyMode_TakeDependency)
}

func (gcs *GenerationalNBS) getManyCompressed(ctx context.Context, hashes hash.HashSet, found func(context.Context, CompressedChunk), gcDepMode gcDependencyMode) error {
var mu sync.Mutex
notInOldGen := hashes.Copy()
err := gcs.oldGen.GetManyCompressed(ctx, hashes, func(ctx context.Context, chunk CompressedChunk) {
err := gcs.oldGen.getManyCompressed(ctx, hashes, func(ctx context.Context, chunk CompressedChunk) {
mu.Lock()
delete(notInOldGen, chunk.Hash())
mu.Unlock()
found(ctx, chunk)
})
}, gcDepMode)
if err != nil {
return err
}
@@ -161,12 +165,12 @@ func (gcs *GenerationalNBS) GetManyCompressed(ctx context.Context, hashes hash.H
}

notFound := notInOldGen.Copy()
err = gcs.newGen.GetManyCompressed(ctx, notInOldGen, func(ctx context.Context, chunk CompressedChunk) {
err = gcs.newGen.getManyCompressed(ctx, notInOldGen, func(ctx context.Context, chunk CompressedChunk) {
mu.Lock()
delete(notFound, chunk.Hash())
mu.Unlock()
found(ctx, chunk)
})
}, gcDepMode)
if err != nil {
return err
}
@@ -176,7 +180,7 @@ func (gcs *GenerationalNBS) GetManyCompressed(ctx context.Context, hashes hash.H

// The missing chunks may be ghost chunks.
if gcs.ghostGen != nil {
return gcs.ghostGen.GetManyCompressed(ctx, notFound, found)
return gcs.ghostGen.getManyCompressed(ctx, notFound, found, gcDepMode)
}
return nil
}
4 changes: 4 additions & 0 deletions go/store/nbs/ghost_store.go
Original file line number Diff line number Diff line change
@@ -91,6 +91,10 @@ func (g GhostBlockStore) GetMany(ctx context.Context, hashes hash.HashSet, found
}

func (g GhostBlockStore) GetManyCompressed(ctx context.Context, hashes hash.HashSet, found func(context.Context, CompressedChunk)) error {
return g.getManyCompressed(ctx, hashes, found, gcDependencyMode_TakeDependency)
}

func (g GhostBlockStore) getManyCompressed(ctx context.Context, hashes hash.HashSet, found func(context.Context, CompressedChunk), gcDepMode gcDependencyMode) error {
for h := range hashes {
if g.skippedRefs.Has(h) {
found(ctx, NewGhostCompressedChunk(h))
47 changes: 35 additions & 12 deletions go/store/nbs/store.go
Original file line number Diff line number Diff line change
@@ -89,6 +89,16 @@ type NBSCompressedChunkStore interface {
GetManyCompressed(context.Context, hash.HashSet, func(context.Context, CompressedChunk)) error
}

type gcDependencyMode int
const (
gcDependencyMode_TakeDependency gcDependencyMode = iota
gcDependencyMode_NoDependency
)

type CompressedChunkStoreForGC interface {
getManyCompressed(context.Context, hash.HashSet, func(context.Context, CompressedChunk), gcDependencyMode) error
}

type NomsBlockStore struct {
mm manifestManager
p tablePersister
@@ -941,22 +951,31 @@ func (nbs *NomsBlockStore) Get(ctx context.Context, h hash.Hash) (chunks.Chunk,
func (nbs *NomsBlockStore) GetMany(ctx context.Context, hashes hash.HashSet, found func(context.Context, *chunks.Chunk)) error {
ctx, span := tracer.Start(ctx, "nbs.GetMany", trace.WithAttributes(attribute.Int("num_hashes", len(hashes))))
defer span.End()
return nbs.getManyWithFunc(ctx, hashes, func(ctx context.Context, cr chunkReader, eg *errgroup.Group, reqs []getRecord, keeper keeperF, stats *Stats) (bool, gcBehavior, error) {
return cr.getMany(ctx, eg, reqs, found, keeper, nbs.stats)
})
return nbs.getManyWithFunc(ctx, hashes, gcDependencyMode_TakeDependency,
func(ctx context.Context, cr chunkReader, eg *errgroup.Group, reqs []getRecord, keeper keeperF, stats *Stats) (bool, gcBehavior, error) {
return cr.getMany(ctx, eg, reqs, found, keeper, nbs.stats)
},
)
}

func (nbs *NomsBlockStore) GetManyCompressed(ctx context.Context, hashes hash.HashSet, found func(context.Context, CompressedChunk)) error {
return nbs.getManyCompressed(ctx, hashes, found, gcDependencyMode_TakeDependency)
}

func (nbs *NomsBlockStore) getManyCompressed(ctx context.Context, hashes hash.HashSet, found func(context.Context, CompressedChunk), gcDepMode gcDependencyMode) error {
ctx, span := tracer.Start(ctx, "nbs.GetManyCompressed", trace.WithAttributes(attribute.Int("num_hashes", len(hashes))))
defer span.End()
return nbs.getManyWithFunc(ctx, hashes, func(ctx context.Context, cr chunkReader, eg *errgroup.Group, reqs []getRecord, keeper keeperF, stats *Stats) (bool, gcBehavior, error) {
return cr.getManyCompressed(ctx, eg, reqs, found, keeper, nbs.stats)
})
return nbs.getManyWithFunc(ctx, hashes, gcDepMode,
func(ctx context.Context, cr chunkReader, eg *errgroup.Group, reqs []getRecord, keeper keeperF, stats *Stats) (bool, gcBehavior, error) {
return cr.getManyCompressed(ctx, eg, reqs, found, keeper, nbs.stats)
},
)
}

func (nbs *NomsBlockStore) getManyWithFunc(
ctx context.Context,
hashes hash.HashSet,
gcDepMode gcDependencyMode,
getManyFunc func(ctx context.Context, cr chunkReader, eg *errgroup.Group, reqs []getRecord, keeper keeperF, stats *Stats) (bool, gcBehavior, error),
) error {
if len(hashes) == 0 {
@@ -976,8 +995,12 @@ func (nbs *NomsBlockStore) getManyWithFunc(
eg.SetLimit(ioParallelism)

nbs.mu.Lock()
keeper := nbs.keeperFunc
if gcDepMode == gcDependencyMode_NoDependency {
keeper = nil
}
if nbs.mt != nil {
remaining, gcb, err := getManyFunc(ctx, nbs.mt, eg, reqs, nbs.keeperFunc, nbs.stats)
remaining, gcb, err := getManyFunc(ctx, nbs.mt, eg, reqs, keeper, nbs.stats)
if err != nil {
nbs.mu.Unlock()
return err
@@ -995,7 +1018,7 @@ func (nbs *NomsBlockStore) getManyWithFunc(
return nil
}
}
tables, keeper, endRead := nbs.tables, nbs.keeperFunc, nbs.beginRead()
tables, endRead := nbs.tables, nbs.beginRead()
nbs.mu.Unlock()

_, gcb, err := getManyFunc(ctx, tables, eg, reqs, keeper, nbs.stats)
@@ -1755,7 +1778,7 @@ func (nbs *NomsBlockStore) MarkAndSweepChunks(ctx context.Context, getAddrs chun
return markAndSweepChunks(ctx, nbs, nbs, dest, getAddrs, filter, mode)
}

func markAndSweepChunks(ctx context.Context, nbs *NomsBlockStore, src NBSCompressedChunkStore, dest chunks.ChunkStore, getAddrs chunks.GetAddrsCurry, filter chunks.HasManyFunc, mode chunks.GCMode) (chunks.MarkAndSweeper, error) {
func markAndSweepChunks(ctx context.Context, nbs *NomsBlockStore, src CompressedChunkStoreForGC, dest chunks.ChunkStore, getAddrs chunks.GetAddrsCurry, filter chunks.HasManyFunc, mode chunks.GCMode) (chunks.MarkAndSweeper, error) {
ops := nbs.SupportedOperations()
if !ops.CanGC || !ops.CanPrune {
return nil, chunks.ErrUnsupportedOperation
@@ -1823,7 +1846,7 @@ func markAndSweepChunks(ctx context.Context, nbs *NomsBlockStore, src NBSCompres
}

type markAndSweeper struct {
src NBSCompressedChunkStore
src CompressedChunkStoreForGC
dest *NomsBlockStore
getAddrs chunks.GetAddrsCurry
filter chunks.HasManyFunc
@@ -1869,7 +1892,7 @@ func (i *markAndSweeper) SaveHashes(ctx context.Context, hashes []hash.Hash) err

found := 0
var addErr error
err = i.src.GetManyCompressed(ctx, toVisit, func(ctx context.Context, cc CompressedChunk) {
err = i.src.getManyCompressed(ctx, toVisit, func(ctx context.Context, cc CompressedChunk) {
mu.Lock()
defer mu.Unlock()
if addErr != nil {
@@ -1893,7 +1916,7 @@ func (i *markAndSweeper) SaveHashes(ctx context.Context, hashes []hash.Hash) err
return
}
addErr = i.getAddrs(c)(ctx, nextToVisit, func(h hash.Hash) bool { return false })
})
}, gcDependencyMode_NoDependency)
if err != nil {
return err
}