Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zoekt: implement mode which has same behaviour as attribution search #613

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

keegancsmith
Copy link
Member

This is to help reproduce slow attribution searches we have on sourcegraph.com.

$ echo -n "const data: Event = { ...JSON.parse(message.data), type: message.event }" | go run ./cmd/zoekt -v -exact_stdin -index_dir ~/zoekt-dotcom-subset-2023-06-29 -cpu_profile profile.cpu
2023/07/13 09:54:26 loading 854 shard(s): github.com%2F214140846%2FTOGO_School_Miniprograme_v16.00000.zoekt, github.com%2F2fd%2Fgraphdoc_v16.00000.zoekt, github.com%2F5t3ph%2Fa11y-color-tokens_v16.00000.zoekt, github.com%2F97-things%2F97-things-every-programmer-should-know_v16.00000.zoekt, github.com%2FADmad%2Fcakephp-jwt-auth_v16.00000.zoekt... 849 more
2023/07/13 09:54:26 query: case_content_substr:"const data: Event = { ...JSON.parse(message.data), type: message.event }"
2023/07/13 09:54:26 Displaying matches...
client/cody-shared/src/sourcegraph-api/completions/browserClient.ts:48:                const data: Event = { ...JSON.parse(message.data), type: message.event }
2023/07/13 09:54:27 stats: zoekt.Stats{ContentBytesLoaded:81980, IndexBytesLoaded:441, Crashes:0, Duration:1481452, FileCount:1, ShardFilesConsidered:0, FilesConsidered:5, FilesLoaded:5, FilesSkipped:0, ShardsScanned:12, ShardsSkipped:0, ShardsSkippedFilter:841, MatchCount:1, NgramMatches:5, NgramLookups:11954, Wait:2860, RegexpsConsidered:0, FlushReason:0x0}

This is to help reproduce slow attribution searches we have on
sourcegraph.com.
Useful in local testing to capture both on and off cpu time spent.
Should consider shipping this in the webserver as well.

Test Plan: ran zoekt with -full_profile flag and inspected output in
pprof.
We believe this will improve performance of the btree lookups.

Test Plan: go test ./...
keys only has a length of 50, so might not be faster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant