Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expose stream-ordering in subword tokenizer API #17206

Merged
merged 6 commits into from
Nov 4, 2024
Merged

Expose stream-ordering in subword tokenizer API #17206

merged 6 commits into from
Nov 4, 2024

Conversation

shrshi
Copy link
Contributor

@shrshi shrshi commented Oct 30, 2024

Description

Add stream parameter to public APIs:

nvtext::subword_tokenize
nvtext::load_vocabulary_file

Added stream gtest.

Reference: #13744

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@github-actions github-actions bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue labels Oct 30, 2024
@shrshi shrshi added strings strings issues (C++ and Python) improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Oct 30, 2024
@shrshi shrshi marked this pull request as ready for review October 30, 2024 00:48
@shrshi shrshi requested a review from a team as a code owner October 30, 2024 00:48
mhaseeb123
mhaseeb123 requested changes Oct 30, 2024
View reviewed changes
Copy link
Member

@mhaseeb123 mhaseeb123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor nits

cpp/tests/streams/text/subword_tokenize_test.cpp Outdated Show resolved Hide resolved
cpp/tests/streams/text/subword_tokenize_test.cpp Outdated Show resolved Hide resolved
cpp/tests/streams/text/subword_tokenize_test.cpp Show resolved Hide resolved
cpp/tests/streams/text/subword_tokenize_test.cpp Outdated Show resolved Hide resolved
cpp/tests/streams/text/subword_tokenize_test.cpp Outdated Show resolved Hide resolved
@shrshi shrshi requested a review from mhaseeb123 October 31, 2024 03:47
@mhaseeb123 mhaseeb123 removed their assignment Oct 31, 2024
@shrshi shrshi mentioned this pull request Oct 31, 2024
Open
@shrshi
Copy link
Contributor Author

shrshi commented Nov 4, 2024

/merge

@rapids-bot rapids-bot bot merged commit 0d37506 into rapidsai:branch-24.12 Nov 4, 2024
102 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davidwendt davidwendt davidwendt approved these changes

@mhaseeb123 mhaseeb123 mhaseeb123 approved these changes

@ttnghia ttnghia ttnghia approved these changes

@mythrocks mythrocks Awaiting requested review from mythrocks mythrocks is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@shrshi shrshi

Labels
CMake CMake build issue improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change strings strings issues (C++ and Python)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shrshi @ttnghia @mhaseeb123 @davidwendt