Skip to content

Commit

Permalink
XTC: simplified and reworked to be closer to original
Browse files Browse the repository at this point in the history
* iterating through candidates from back to front makes it easy to keep the last of top tokens
* some important commits from llama.cpp
  • Loading branch information
MaggotHATE committed Aug 26, 2024
1 parent 6b69d0b commit 20b6827
Show file tree
Hide file tree
Showing 6 changed files with 148,978 additions and 148,971 deletions.
16 changes: 11 additions & 5 deletions base/common.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -182,13 +182,19 @@ int get_math_cpu_count() {

void string_replace_all(std::string & s, const std::string & search, const std::string & replace) {
if (search.empty()) {
return; // Avoid infinite loop if 'search' is an empty string
return;
}
std::string builder;
builder.reserve(s.length());
size_t pos = 0;
while ((pos = s.find(search, pos)) != std::string::npos) {
s.replace(pos, search.length(), replace);
pos += replace.length();
}
size_t last_pos = 0;
while ((pos = s.find(search, last_pos)) != std::string::npos) {
builder.append(s, last_pos, pos - last_pos);
builder.append(replace);
last_pos = pos + search.length();
}
builder.append(s, last_pos, std::string::npos);
s = std::move(builder);
}

void process_escapes(std::string& input) {
Expand Down
Loading

0 comments on commit 20b6827

Please sign in to comment.