Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exclude the comments in the tokenization step #1645

Merged
merged 6 commits into from
Nov 20, 2024
Merged

Exclude the comments in the tokenization step #1645

merged 6 commits into from
Nov 20, 2024

Conversation

milachae
Copy link
Collaborator

By default, comments will now be excluded during the tokenization of a file.

I also added a CLI option (-ic or --include-comments) to include the comments during tokenization.

@milachae milachae requested a review from rien November 19, 2024 13:39
@milachae milachae self-assigned this Nov 19, 2024
rien
rien requested changes Nov 20, 2024
View reviewed changes
Copy link
Member

@rien rien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test to check if setting includeComments = true actually includes the comments?

cli/src/cli/commands/run.ts Outdated Show resolved Hide resolved
lib/src/lib/tokenizer/tokenizer.ts Outdated Show resolved Hide resolved
lib/src/test/tokenizer.test.ts Outdated Show resolved Hide resolved
lib/src/lib/tokenizer/codeTokenizer.ts Outdated Show resolved Hide resolved
lib/src/lib/tokenizer/codeTokenizer.ts Outdated Show resolved Hide resolved
@milachae milachae requested a review from rien November 20, 2024 10:16
@rien rien added this pull request to the merge queue Nov 20, 2024
Merged via the queue into main with commit a8ac373 Nov 20, 2024
26 checks passed
@rien rien deleted the exclude-comments branch November 20, 2024 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rien rien rien approved these changes

Assignees

@milachae milachae

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@milachae @rien