Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] main from dolthub:main #54

Merged
merged 110 commits into from
Feb 8, 2025
Merged
Changes from 1 commit
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
3c1a365
go/store/nbs: store.go: Add a mechanism to bracket outstanding read r…
reltuk Jan 15, 2025
f099fdc
go/store/nbs: chunkReader,chunkSource: GC: Add the ability to take de…
reltuk Jan 15, 2025
26c909d
go/store/nbs: Add GC keeper calls on reads through NomsBlockStore.
reltuk Jan 16, 2025
4a59dcb
go/store/nbs: tablePersister: Add GC dependency capturing to written …
reltuk Jan 16, 2025
78ee20a
go/store/nbs: table_set: append: Thread GC dependency tracking throug…
reltuk Jan 16, 2025
e423a5c
go/store/nbs: store.go: MarkAndSweepChunks: After adding read trackin…
reltuk Jan 16, 2025
6ad9b37
repofmt.sh.
reltuk Jan 16, 2025
5355928
Avoid loading DB for commands where it's not necessary.
nicktobey Jan 22, 2025
1d4a82f
Make dEnv.DoltDB private and add an accessor function.
nicktobey Jan 23, 2025
8dfadf9
Propagate context.
nicktobey Jan 23, 2025
9c04d50
Remove explicit list of commands that not need to eagerly load the DB.
nicktobey Jan 23, 2025
a963a85
Don't load the database until we actually bind the queryist.
nicktobey Jan 23, 2025
b6c233d
Remove explicit calls to `dEnv.ReloadDB()` from commands.
nicktobey Jan 23, 2025
e89385d
Add url parameter to LoadWithoutDB and remove LoadWithDeferredDB
nicktobey Jan 23, 2025
d43ce7f
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 23, 2025
849e5c0
Create context object for remote server main function.
nicktobey Jan 23, 2025
e568a66
Fix infinite recursion loop in DoltEnv.DoltDB()
nicktobey Jan 23, 2025
5c04d5f
go/store/nbs: store.go: Fix errgroup context usage-after-Wait bug in …
reltuk Jan 23, 2025
d0fb77f
Fixup tests.
nicktobey Jan 23, 2025
7f0ca8c
Respond to PR feedback.
nicktobey Jan 23, 2025
411b676
Respond to PR feedback.
nicktobey Jan 24, 2025
e540aff
Add context variable to tests.
nicktobey Jan 24, 2025
f78cc0d
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 24, 2025
8b10e7f
Update bats tests to assert that you can't clone directly from anothe…
nicktobey Jan 25, 2025
b830fbc
[ga-bump-dep] Bump dependency in Dolt by jycor (#8790)
coffeegoddd Jan 27, 2025
2d3105e
Add the dirty column to the dolt_branches system table
macneale4 Jan 27, 2025
809be27
Tests for the dolt_branches dirty column
macneale4 Jan 27, 2025
9da62eb
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
macneale4 Jan 27, 2025
fdc79a5
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 27, 2025
9b9722f
Avoid race conditions when loading dolt DB.
nicktobey Jan 27, 2025
5bf5642
Fix workbench tests to expect dirty column
macneale4 Jan 27, 2025
da7bb5c
Don't error out when no workingset is found
macneale4 Jan 27, 2025
fad9c1d
Merge pull request #8793 from dolthub/macneale4/dirty-branch-column
macneale4 Jan 27, 2025
bde0bf9
[ga-bump-release] Update Dolt version to 1.48.0 and release v1.48.0
coffeegoddd Jan 28, 2025
003e2d1
go: binlogreplication: Add Session{{Begin,End}Command,End} lifecycle …
reltuk Jan 28, 2025
89eb2af
go: cmd/dolt: sqlengine: Actually call DoltBinlogReplicaController.Cl…
reltuk Jan 28, 2025
4895747
[ga-bump-dep] Bump dependency in Dolt by zachmu
zachmu Jan 28, 2025
78e9a8a
Merge pull request #8760 from dolthub/aaron/gc-read-dependencies
reltuk Jan 28, 2025
1b59420
go/store/nbs: generational_chunk_store.go: In GCMode_Full, also take …
reltuk Jan 28, 2025
5b55d32
go: sqlserver: Clean up how remotesrv gets a handle to the mrEnv.File…
reltuk Jan 28, 2025
a6b1a26
go: sqle,remotesrv: Implement sql.Session lifecycle callbacks for sql…
reltuk Jan 28, 2025
c401029
Merge pull request #8794 from dolthub/aaron/sql-replication-session-l…
reltuk Jan 28, 2025
366e466
go: sqle/dprocedures: dolt_gc: Move to an intantiated instance.
reltuk Jan 21, 2025
915e392
go: sqle: dsess: Make DoltSession Lifecycle aware. Move towards a GCS…
reltuk Jan 22, 2025
ef954d0
go: sqle/dprocedures: dolt_gc: Implement a session aware safepoint co…
reltuk Jan 23, 2025
bdc8ff1
dolt_gc,dsess: Add VisitGCRoots to dsess.Session and call it from dol…
reltuk Jan 24, 2025
fc3217e
go: sqle: dolt_gc: Add DOLT_GC_SAFEPOINT_CONTROLLER_CHOICE env variab…
reltuk Jan 28, 2025
13c9ddf
go: sqlserver,binlogreplication: Clean up session usage a little to h…
reltuk Jan 29, 2025
caaa4bd
Merge pull request #8795 from dolthub/zachmu-3f5bb8c5
Hydrocharged Jan 29, 2025
819136d
go: dtables: help_table: Manually specify that dolt_gc exists for now.
reltuk Jan 29, 2025
d27b998
prevent nil panic on DoltEnv::loadDBOnce
nicktobey Jan 29, 2025
cda466f
/go/libraries/doltcore/env/actions: make iter resolved tags paginated…
coffeegoddd Jan 21, 2025
09dd814
/go/libraries/doltcore/env/actions/tag.go: reinstage iterresolved tag…
coffeegoddd Jan 30, 2025
8356ba7
go: store/types: Fix dolt_gc on databases that use vector indexes.
reltuk Jan 30, 2025
3dd50f5
Merge pull request #8796 from dolthub/aaron/gc-read-dependencies-oldg…
reltuk Jan 30, 2025
ddaa922
Fix typo in error message.
reltuk Jan 30, 2025
337671a
Merge pull request #8774 from dolthub/db/tags-3
coffeegoddd Jan 30, 2025
153d46b
go: sqle: gc_safepoint_controller: PR feedback, comments to explain s…
reltuk Jan 30, 2025
5826451
Merge remote-tracking branch 'origin/main' into aaron/dsess-lifecycle
reltuk Jan 30, 2025
7c1cff7
integration-tests/go-sql-server-driver: Fix bug in concurrent_gc_test.
reltuk Jan 30, 2025
f79db4a
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
reltuk Jan 30, 2025
b9f6a56
go: sqle: dprocedures: dolt_gc: Get the safepoint controller from the…
reltuk Jan 30, 2025
b1aa361
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Jan 30, 2025
9fd5d56
[ga-bump-dep] Bump dependency in Dolt by Hydrocharged
Hydrocharged Jan 31, 2025
4a2606e
Merge pull request #8803 from dolthub/Hydrocharged-67aa2a43
Hydrocharged Jan 31, 2025
3821a7a
Include deleteErr message in returned error.
fulghum Jan 31, 2025
84bd03d
Merge pull request #8804 from dolthub/fulghum/delete-err
fulghum Jan 31, 2025
16ae96b
[no-release-notes] go: binlogreplication: tests: Parallelize the test…
reltuk Jan 31, 2025
f33c9fc
Merge pull request #8801 from dolthub/aaron/ivff-gc
reltuk Jan 31, 2025
0ef447e
Merge pull request #8797 from dolthub/aaron/sql-server-remotesapi-ses…
reltuk Jan 31, 2025
f131fb8
go/libraries/doltcore/sqle/binlogreplication: Fix windows build tags.
reltuk Jan 31, 2025
ed1af8f
Merge pull request #8805 from dolthub/aaron/binlog-tests-parallel
reltuk Jan 31, 2025
52b625f
Merge remote-tracking branch 'origin/main' into aaron/dsess-lifecycle
reltuk Feb 3, 2025
b44469d
go: sqle/remotesrv.go: Fix session lifecycle callbacks interceptors.
reltuk Feb 3, 2025
4e6378f
Merge pull request #8798 from dolthub/aaron/dsess-lifecycle
reltuk Feb 3, 2025
3d5d419
Update tag_test.go
nicktobey Feb 3, 2025
2911085
Merge remote-tracking branch 'origin/main' into nicktobey/lazy-load
nicktobey Feb 3, 2025
3cf7df2
print start server log
liuliu-dev Feb 3, 2025
9dd401c
Deprecate user and pass options for sql-server
fulghum Jan 30, 2025
00cacf4
printf
liuliu-dev Feb 3, 2025
d1e90c4
Merge pull request #8806 from dolthub/liuliu/replace-cli-printErrf-wi…
liuliu-dev Feb 3, 2025
b4b6d71
Merge pull request #8783 from dolthub/nicktobey/lazy-load
nicktobey Feb 3, 2025
6d95550
go: sqle/binlogreplication: Add some waits for more reliable observat…
reltuk Feb 4, 2025
55001ef
Tidying up journal writer code comments and error handling
fulghum Jan 10, 2025
c0c9c64
PR Feedback: Making processJournalRecords() less aggressive on error'…
fulghum Jan 29, 2025
e701e5e
[no-release-notes] go: nbs/store: Small cleanups to iterateAllChunks.
reltuk Feb 4, 2025
56b42c9
[no-release-notes] go: store/nbs: GhostBlockStore: Add refCheck which…
reltuk Feb 4, 2025
04498d5
Merge pull request #8732 from dolthub/fulghum/journal
fulghum Feb 4, 2025
13b81b9
[no-release-notes] go: store/nbs: Unify logic from UpdateManifest and…
reltuk Feb 4, 2025
094fc7b
[kvexec] fix left join nullability bug (#8812)
max-hoffman Feb 4, 2025
3fa6a29
[kvexec] literal types cast to lookup expr type (#8808)
max-hoffman Feb 4, 2025
928ae32
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
reltuk Feb 4, 2025
a279a81
go: store/nbs: ghost_store.go: Fix bug in refCheck so it actually upd…
reltuk Feb 5, 2025
4398e55
Bug fix for encoding extended types in keys
zachmu Feb 2, 2025
9a2e242
[ga-format-pr] Run go/utils/repofmt/format_repo.sh and go/Godeps/upda…
zachmu Feb 5, 2025
e81bed9
[ga-bump-dep] Bump dependency in Dolt by max-hoffman (#8814)
coffeegoddd Feb 5, 2025
d766206
Merge pull request #8800 from dolthub/fulghum/rm-sql-server-user
fulghum Feb 5, 2025
5dece74
Merge pull request #8809 from dolthub/aaron/binlog-replica-test-sleeps
reltuk Feb 5, 2025
568590e
Merge pull request #8811 from dolthub/aaron/nbs-ghost-store-refcheck
reltuk Feb 5, 2025
8cc323e
[ga-bump-release] Update Dolt version to 1.49.0 and release v1.49.0
coffeegoddd Feb 5, 2025
cb155c5
Merge pull request #8810 from dolthub/aaron/nbs-iterate-all-chunks-stats
reltuk Feb 5, 2025
973ca47
Merge pull request #8813 from dolthub/aaron/nbs-manifest-update-dedup
reltuk Feb 5, 2025
cc43725
fix build break (#8819)
macneale4 Feb 5, 2025
9e36f51
Merge pull request #8817 from dolthub/zachmu/extended-encoding
zachmu Feb 5, 2025
edfa9c7
Expand itemWidth field to 32 bits.
nicktobey Feb 7, 2025
34f9158
Revert PR #8723 which disabled the smart JSON chunker when the docume…
nicktobey Feb 7, 2025
c6d7f65
Re-enable disabled json test.
nicktobey Feb 7, 2025
f703df9
Merge pull request #8831 from dolthub/nicktobey/jsonread
nicktobey Feb 7, 2025
5b028a5
[no-release-notes] fix bad tests and bump (#8837)
jycor Feb 7, 2025
13d1c2d
[ga-bump-release] Update Dolt version to 1.49.1 and release v1.49.1
coffeegoddd Feb 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
go/store/nbs: Add GC keeper calls on reads through NomsBlockStore.
  • Loading branch information
reltuk committed Jan 16, 2025
commit 26c909d28b205f6bb854e975c57fe83fa68faae4
5 changes: 4 additions & 1 deletion go/store/nbs/archive_chunk_source.go
Original file line number Diff line number Diff line change
@@ -106,7 +106,10 @@ func (acs archiveChunkSource) getMany(ctx context.Context, eg *errgroup.Group, r
for i, req := range reqs {
h := *req.a
data, err := acs.aRdr.get(h)
if err != nil || data == nil {
if err != nil {
return true, gcBehavior_Continue, err
}
if data == nil {
foundAll = false
} else {
if keeper != nil && keeper(h) {
45 changes: 31 additions & 14 deletions go/store/nbs/generational_chunk_store.go
Original file line number Diff line number Diff line change
@@ -118,7 +118,9 @@ func (gcs *GenerationalNBS) GetMany(ctx context.Context, hashes hash.HashSet, fo
return nil
}

err = gcs.newGen.GetMany(ctx, notFound, func(ctx context.Context, chunk *chunks.Chunk) {
hashes = notFound
notFound = hashes.Copy()
err = gcs.newGen.GetMany(ctx, hashes, func(ctx context.Context, chunk *chunks.Chunk) {
func() {
mu.Lock()
defer mu.Unlock()
@@ -202,14 +204,30 @@ func (gcs *GenerationalNBS) Has(ctx context.Context, h hash.Hash) (bool, error)
}

// HasMany returns a new HashSet containing any members of |hashes| that are absent from the store.
func (gcs *GenerationalNBS) HasMany(ctx context.Context, hashes hash.HashSet) (absent hash.HashSet, err error) {
gcs.newGen.mu.RLock()
defer gcs.newGen.mu.RUnlock()
return gcs.hasMany(toHasRecords(hashes))
func (gcs *GenerationalNBS) HasMany(ctx context.Context, hashes hash.HashSet) (hash.HashSet, error) {
absent, err := gcs.newGen.HasMany(ctx, hashes)
if err != nil {
return nil, err
}
if len(absent) == 0 {
return nil, err
}

absent, err = gcs.oldGen.HasMany(ctx, absent)
if err != nil {
return nil, err
}
if len(absent) == 0 || gcs.ghostGen == nil {
return nil, err
}

return gcs.ghostGen.HasMany(ctx, absent)
}

func (gcs *GenerationalNBS) hasMany(recs []hasRecord) (absent hash.HashSet, err error) {
absent, err = gcs.newGen.hasMany(recs)
// |refCheck| is called from write processes in newGen, so it is called with
// newGen.mu held. oldGen.mu is not held however.
func (gcs *GenerationalNBS) refCheck(recs []hasRecord) (hash.HashSet, error) {
absent, err := gcs.newGen.refCheck(recs)
if err != nil {
return nil, err
} else if len(absent) == 0 {
@@ -219,12 +237,11 @@ func (gcs *GenerationalNBS) hasMany(recs []hasRecord) (absent hash.HashSet, err
absent, err = func() (hash.HashSet, error) {
gcs.oldGen.mu.RLock()
defer gcs.oldGen.mu.RUnlock()
return gcs.oldGen.hasMany(recs)
return gcs.oldGen.refCheck(recs)
}()
if err != nil {
return nil, err
}

if len(absent) == 0 || gcs.ghostGen == nil {
return absent, nil
}
@@ -237,7 +254,7 @@ func (gcs *GenerationalNBS) hasMany(recs []hasRecord) (absent hash.HashSet, err
// to Flush(). Put may be called concurrently with other calls to Put(),
// Get(), GetMany(), Has() and HasMany().
func (gcs *GenerationalNBS) Put(ctx context.Context, c chunks.Chunk, getAddrs chunks.GetAddrsCurry) error {
return gcs.newGen.putChunk(ctx, c, getAddrs, gcs.hasMany)
return gcs.newGen.putChunk(ctx, c, getAddrs, gcs.refCheck)
}

// Returns the NomsBinFormat with which this ChunkSource is compatible.
@@ -277,7 +294,7 @@ func (gcs *GenerationalNBS) Root(ctx context.Context) (hash.Hash, error) {
// persisted root hash from last to current (or keeps it the same).
// If last doesn't match the root in persistent storage, returns false.
func (gcs *GenerationalNBS) Commit(ctx context.Context, current, last hash.Hash) (bool, error) {
return gcs.newGen.commit(ctx, current, last, gcs.hasMany)
return gcs.newGen.commit(ctx, current, last, gcs.refCheck)
}

// Stats may return some kind of struct that reports statistics about the
@@ -400,18 +417,18 @@ func (gcs *GenerationalNBS) AddTableFilesToManifest(ctx context.Context, fileIdT

// PruneTableFiles deletes old table files that are no longer referenced in the manifest of the new or old gen chunkstores
func (gcs *GenerationalNBS) PruneTableFiles(ctx context.Context) error {
err := gcs.oldGen.pruneTableFiles(ctx, gcs.hasMany)
err := gcs.oldGen.pruneTableFiles(ctx)

if err != nil {
return err
}

return gcs.newGen.pruneTableFiles(ctx, gcs.hasMany)
return gcs.newGen.pruneTableFiles(ctx)
}

// SetRootChunk changes the root chunk hash from the previous value to the new root for the newgen cs
func (gcs *GenerationalNBS) SetRootChunk(ctx context.Context, root, previous hash.Hash) error {
return gcs.newGen.setRootChunk(ctx, root, previous, gcs.hasMany)
return gcs.newGen.setRootChunk(ctx, root, previous, gcs.refCheck)
}

// SupportedOperations returns a description of the support TableFile operations. Some stores only support reading table files, not writing.
Loading