The output of the main service is inconsistent with that of the server service #6569
Comments
|
@auroralsun If you do the effort to describe your issue in english I might try to help |
auroralsun
changed the title
Thank you for your reply. I have tried to redescribe the corresponding situation in English. I am looking forward to your answer. |
|
@auroralsun for me, it's not clear how to Reproduce the issue. Your issue may be valid, but it's hard to say the exact cause of the issue(because your examples are in Chinese). Are you able to Reproduce the issue in English, or is it a language-specific problem? Is there an English example or test that clearly shows the difference in output for |
It could be a language-specific problem. This is a problem found in the application at present, and there are not many cases involving English, I will try my best to find English examples. Thank you for your answer. |
|
@auroralsun ideally if you are able to submit a server test scenario in Chinese, this would be awesome. See the server test folder, cucumber based. |
phymbert
added
need more info
When I tried to test it on the open source model chinese-llama 2-7b.gguf ( https://huggingface.co/shaowenchen/chinese-llama-2-7b-gguf ), I also had this problem in the server service. When I input the whole Chinese word for BYD ("比亚迪"), the model returns to me a feeling I didn't get, but correctly returns when there is a noisy space in the middle (such as "比 亚迪" or "比亚 迪"). In addition, try to load a quantitative model with ctransformer for these words to get the same expected answers as the main service. |
|
Looks like an issue with the bpe tokenizer. @ggerganov, are you aware of a better workaround |
|
It could be related to the tokenize pre-processing. Don't think we have workaround - we need to finish #5981 |
|
This issue was closed because it has been inactive for 14 days since being marked as stale. |
When the same quantitative model is used for server service and main service, some specific words are answered differently. It seems that the input specific words are not received or received incorrectly.
For example, BYD, Tesla, Lexus and other car names have this problem, such as Geely, BMW, Audi and so on is normal.
The specific problem is manifested in: When obtaining the word "BYD" in the server service, non-Chinese characters such as "ruit" are not obtained or obtained. As in the first example, when asked about BYD car, the reply only involved the car, and BYD was lost.
Test results in the server
These are three examples of problems（BYD）
{
content: ' 汽车是一种交通工具，它通常由发动机，变速箱，底盘和底盘系统，悬挂系统，转向系统，车身和车轮等组成。汽车通常由汽油或柴油发动机提供动力，通过变速箱和传动系统来控制车辆行驶的速度和方向。汽车的设计和制造技术不断提高，汽车的功能也越来越强大。现在汽车已经不仅仅是一种交通工具，它已经成为人们日常生活中不可或缺的一部分，提供了各种便利。汽车在现代社会中的作用非常广泛，它可以满足人们的出行需求，同时也可以娱乐休闲，提高生活质量。汽车的使用也带来了一些负面影响，如空气污染，交通拥堵，交通事故等。因此，汽车的使用应该更加理性，更加安全，更加环保。',
id_slot: 0,
stop: true,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
tokens_predicted: 153,
tokens_evaluated: 40,
generation_settings: {
n_ctx: 4096,
n_predict: -1,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
seed: 4294967295,
temperature: 0.800000011920929,
dynatemp_range: 0,
dynatemp_exponent: 1,
top_k: 40,
top_p: 0.949999988079071,
min_p: 0.05000000074505806,
tfs_z: 1,
typical_p: 1,
repeat_last_n: 64,
repeat_penalty: 1,
presence_penalty: 0,
frequency_penalty: 0,
penalty_prompt_tokens: [],
use_penalty_prompt_tokens: false,
mirostat: 0,
mirostat_tau: 5,
mirostat_eta: 0.10000000149011612,
penalize_nl: false,
stop: [],
n_keep: 1,
n_discard: 0,
ignore_eos: false,
stream: false,
logit_bias: [],
n_probs: 0,
min_keep: 0,
grammar: '',
samplers: [ 'top_k', 'tfs_z', 'typical_p', 'top_p', 'min_p', 'temperature' ]
},
prompt: '[INST] <>\n' +
'You are a helpful assistant. 你是一个乐于助人的助手。\n' +
'<>\n' +
'\n' +
'请详细介绍一下比亚迪汽车[/INST]',
truncated: false,
stopped_eos: true,
stopped_word: false,
stopped_limit: false,
stopping_word: '',
tokens_cached: 192,
timings: {
prompt_n: 40,
prompt_ms: 71.919,
prompt_per_token_ms: 1.7979749999999999,
prompt_per_second: 556.1812594724621,
predicted_n: 153,
predicted_ms: 1859.683,
predicted_per_token_ms: 12.154790849673203,
predicted_per_second: 82.27208615661917
}
}
{
content: ' 很抱歉，但是我无法准确回答您的问题，因为您没有提供任何关于它的信息。我需要知道什么是"ruit"来帮助您。',
id_slot: 0,
stop: true,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
tokens_predicted: 32,
tokens_evaluated: 39,
generation_settings: {
n_ctx: 4096,
n_predict: -1,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
seed: 4294967295,
temperature: 0.800000011920929,
dynatemp_range: 0,
dynatemp_exponent: 1,
top_k: 40,
top_p: 0.949999988079071,
min_p: 0.05000000074505806,
tfs_z: 1,
typical_p: 1,
repeat_last_n: 64,
repeat_penalty: 1,
presence_penalty: 0,
frequency_penalty: 0,
penalty_prompt_tokens: [],
use_penalty_prompt_tokens: false,
mirostat: 0,
mirostat_tau: 5,
mirostat_eta: 0.10000000149011612,
penalize_nl: false,
stop: [],
n_keep: 1,
n_discard: 0,
ignore_eos: false,
stream: false,
logit_bias: [],
n_probs: 0,
min_keep: 0,
grammar: '',
samplers: [ 'top_k', 'tfs_z', 'typical_p', 'top_p', 'min_p', 'temperature' ]
},
prompt: '[INST] <>\n' +
'You are a helpful assistant. 你是一个乐于助人的助手。\n' +
'<>\n' +
'\n' +
'请详细介绍一下比亚迪[/INST]',
truncated: false,
stopped_eos: true,
stopped_word: false,
stopped_limit: false,
stopping_word: '',
tokens_cached: 70,
timings: {
prompt_n: 39,
prompt_ms: 50.617,
prompt_per_token_ms: 1.2978717948717948,
prompt_per_second: 770.4921271509572,
predicted_n: 32,
predicted_ms: 382.638,
predicted_per_token_ms: 11.9574375,
predicted_per_second: 83.629958341827
}
}
{
content: ' 驱逐舰05汽车（Discharged Ship05）是一款由中国长城汽车制造的纯电动紧凑型轿车。它的外观设计灵感来自驱逐舰05系列的驱逐舰，具有较高的辨识度。驱逐舰05汽车采用三元锂离子电池，能够提供高效的续航能力，最大时速可达160公里。它还拥有先进的智能控制系统，能够根据路况和驾驶需求，自动调整车辆的加速、刹车和转向等性能。此外，它的充电时间短，续航能力强，是一款值得购买的纯电动紧凑型轿车。',
id_slot: 0,
stop: true,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
tokens_predicted: 125,
tokens_evaluated: 45,
generation_settings: {
n_ctx: 4096,
n_predict: -1,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
seed: 4294967295,
temperature: 0.800000011920929,
dynatemp_range: 0,
dynatemp_exponent: 1,
top_k: 40,
top_p: 0.949999988079071,
min_p: 0.05000000074505806,
tfs_z: 1,
typical_p: 1,
repeat_last_n: 64,
repeat_penalty: 1,
presence_penalty: 0,
frequency_penalty: 0,
penalty_prompt_tokens: [],
use_penalty_prompt_tokens: false,
mirostat: 0,
mirostat_tau: 5,
mirostat_eta: 0.10000000149011612,
penalize_nl: false,
stop: [],
n_keep: 1,
n_discard: 0,
ignore_eos: false,
stream: false,
logit_bias: [],
n_probs: 0,
min_keep: 0,
grammar: '',
samplers: [ 'top_k', 'tfs_z', 'typical_p', 'top_p', 'min_p', 'temperature' ]
},
prompt: '[INST] <>\n' +
'You are a helpful assistant. 你是一个乐于助人的助手。\n' +
'<>\n' +
'\n' +
'请详细介绍一下比亚迪驱逐舰05汽车[/INST]',
truncated: false,
stopped_eos: true,
stopped_word: false,
stopped_limit: false,
stopping_word: '',
tokens_cached: 169,
timings: {
prompt_n: 45,
prompt_ms: 51.557,
prompt_per_token_ms: 1.1457111111111111,
prompt_per_second: 872.8203735671199,
predicted_n: 125,
predicted_ms: 1518.842,
predicted_per_token_ms: 12.150736,
predicted_per_second: 82.29954136111589
}
}
These are two correct examples（Geely）
{
content: ' 吉利汽车是一家位于中国的汽车制造商，成立于1946年。吉利是一家独立汽车制造商，它生产了许多成功的汽车，包括轿车，SUV，MPV和紧凑型车。\n' +
'\n' +
'吉利汽车在汽车设计和制造方面拥有很高的声誉，它的车型受到许多消费者的喜爱。吉利汽车的品牌形象也得到了提高，在全球范围内都有很好的声誉。\n' +
'\n' +
'吉利汽车的车型包括吉利GS4，吉利GS5，吉利GX7，吉利M6等。这些车型都具有时尚的外观，高质量的内饰和出色的性能。\n' +
'\n' +
'吉利汽车的生产基地遍布于中国各地，其中吉利汽车的总部位于浙江吉利汽车城。吉利汽车一直致力于提高汽车生产技术，并始终保持着对汽车技术的创新和发展。\n' +
'\n' +
'吉利汽车在全球范围内都有广泛的销售，在欧洲，日本和印度都有吉利汽车的销售网络。吉利汽车的目标是通过生产优质的汽车，为人们提供便捷、舒适、安全、经济的交通工具。',
id_slot: 0,
stop: true,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
tokens_predicted: 213,
tokens_evaluated: 40,
generation_settings: {
n_ctx: 4096,
n_predict: -1,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
seed: 4294967295,
temperature: 0.800000011920929,
dynatemp_range: 0,
dynatemp_exponent: 1,
top_k: 40,
top_p: 0.949999988079071,
min_p: 0.05000000074505806,
tfs_z: 1,
typical_p: 1,
repeat_last_n: 64,
repeat_penalty: 1,
presence_penalty: 0,
frequency_penalty: 0,
penalty_prompt_tokens: [],
use_penalty_prompt_tokens: false,
mirostat: 0,
mirostat_tau: 5,
mirostat_eta: 0.10000000149011612,
penalize_nl: false,
stop: [],
n_keep: 1,
n_discard: 0,
ignore_eos: false,
stream: false,
logit_bias: [],
n_probs: 0,
min_keep: 0,
grammar: '',
samplers: [ 'top_k', 'tfs_z', 'typical_p', 'top_p', 'min_p', 'temperature' ]
},
prompt: '[INST] <>\n' +
'You are a helpful assistant. 你是一个乐于助人的助手。\n' +
'<>\n' +
'\n' +
'请详细介绍一下吉利汽车[/INST]',
truncated: false,
stopped_eos: true,
stopped_word: false,
stopped_limit: false,
stopping_word: '',
tokens_cached: 252,
timings: {
prompt_n: 40,
prompt_ms: 67.825,
prompt_per_token_ms: 1.6956250000000002,
prompt_per_second: 589.7530409141173,
predicted_n: 213,
predicted_ms: 2621.52,
predicted_per_token_ms: 12.307605633802817,
predicted_per_second: 81.25057218712809
}
}
{
content: ' 吉利汽车是一家中国汽车品牌，总部位于浙江省杭州市。吉利是一家以轿车、SUV、紧凑型车和小型车为主要产品的公司，同时拥有先进的技术和创新的车型，为消费者提供舒适、安全、时尚且经济实用的汽车。吉利汽车在全球范围内都具有很高的声誉，并以其高性价比和优秀的性能著称。',
id_slot: 0,
stop: true,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
tokens_predicted: 76,
tokens_evaluated: 39,
generation_settings: {
n_ctx: 4096,
n_predict: -1,
model: '../Chinese-LLaMA-Alpaca-2-main/examples/sft_8/ggml-model-f8.gguf',
seed: 4294967295,
temperature: 0.800000011920929,
dynatemp_range: 0,
dynatemp_exponent: 1,
top_k: 40,
top_p: 0.949999988079071,
min_p: 0.05000000074505806,
tfs_z: 1,
typical_p: 1,
repeat_last_n: 64,
repeat_penalty: 1,
presence_penalty: 0,
frequency_penalty: 0,
penalty_prompt_tokens: [],
use_penalty_prompt_tokens: false,
mirostat: 0,
mirostat_tau: 5,
mirostat_eta: 0.10000000149011612,
penalize_nl: false,
stop: [],
n_keep: 1,
n_discard: 0,
ignore_eos: false,
stream: false,
logit_bias: [],
n_probs: 0,
min_keep: 0,
grammar: '',
samplers: [ 'top_k', 'tfs_z', 'typical_p', 'top_p', 'min_p', 'temperature' ]
},
prompt: '[INST] <>\n' +
'You are a helpful assistant. 你是一个乐于助人的助手。\n' +
'<>\n' +
'\n' +
'请详细介绍一下吉利[/INST]',
truncated: false,
stopped_eos: true,
stopped_word: false,
stopped_limit: false,
stopping_word: '',
tokens_cached: 114,
timings: {
prompt_n: 39,
prompt_ms: 74.161,
prompt_per_token_ms: 1.9015641025641026,
prompt_per_second: 525.8828764444924,
predicted_n: 76,
predicted_ms: 922.532,
predicted_per_token_ms: 12.138578947368421,
predicted_per_second: 82.38196615401958
}
}
However, the main service returns correct terms that are recognized.
These are three correct examples（BYD）
请详细介绍一下比亚迪
比亚迪是一家中国品牌，它生产汽车，包括紧凑型轿车、纯电动紧凑型轿车和电动紧凑型轿车。
请详细介绍一下比亚迪汽车
比亚迪是一家著名的中国汽车制造商，总部位于广东省深圳市南山区，成立于1995年1月16日。比亚迪的业务涵盖汽车、电子能源系统、新能源汽车和零配件制造。比亚迪的品牌形象是“勇于创新，致力于改善人们的生活”。
比亚迪的汽车业务始于2000年，并迅速发展成为中国汽车行业的领军者之一。比亚迪的车型包括紧凑型轿车、紧凑型轿车、紧凑型SUV和紧凑型SUV。比亚迪的电动汽车也是全球领先的，其中包括纯电动汽车、插电混动汽车和混合动力汽车。 比亚迪的电子能源系统业务包括电池制造、电池管理系统和电动驱动系统，并提供相关的售后服务。 比亚迪的零配件制造业务包括电池、电机和电动驱动系统。 比亚迪的品牌形象和产品性能受到了广泛的认可和赞赏。比亚迪的电动汽车在全球范围内都取得了巨大的成功。比亚迪一直致力于改善人们的生活，通过创新技术和产品为人类带来更多的便利和舒适。
请详细介绍一下比亚迪驱逐舰05
比亚迪驱逐舰05是一款紧凑型纯电动轿车，由比亚迪集团生产。它于2021年9月正式上市。比亚迪驱逐舰05采用了比亚迪家族化的鹰眼式前脸，大尺寸的前格栅和大尺寸的前灯组使车头显得非常威严。车身侧面采用流畅的线条，车顶微微隆起。车尾采用简洁的设计，采用封闭式尾灯，底部有银色护板。比亚迪驱逐舰05配备了电动助力转向和自动挡变速箱。车辆的悬架采用前双球后双球的独立悬架设计，以确保车辆在行驶过程中更加稳定。比亚迪驱逐舰05配备了电动机，最大输出功率为160千瓦，最大扭矩为252牛米。电池组采用比亚迪自家的电池，续航能力强，在满电状态下可续航500公里。比亚迪驱逐舰05还具有智能驾驶辅助功能，包括主动刹车、盲区监测、车道
These are two correct examples（Geely）
请详细介绍一下吉利
吉利汽车是中国吉利汽车集团旗下品牌。吉利汽车成立于1986年，总部位于浙江省杭州市。吉利汽车是一家以创新、安全、环保和品质为重点的汽车制造商。吉利品牌在全球范围内拥有众多知名车型，包括吉利GX7，吉利GX5，吉利GS8，吉利GX3等。吉利汽车一直致力于为消费者提供卓越的汽车产品，以满足不同消费者的需求。
请详细介绍一下吉利汽车
吉利汽车是吉利汽车集团的品牌之一，总部位于吉利汽车集团的总部位于浙江省杭州市。吉利汽车集团是一家大型的中国汽车制造商，成立于1986年。 吉利汽车集团生产各种类型的汽车，包括轿车，SUV，跨界车，MPV等。吉利汽车集团一直致力于生产高质量，节能，环保的汽车，在亚洲和全球范围内都享有盛誉。 吉利汽车集团的车型包括吉利GX7，吉利M8，吉利GX8，吉利M3，吉利M4，吉利M6，吉利M9，吉利M5，吉利M7，吉利M8L，吉利M10等。
This is the log from the server service
server_log.txt
This is the log from the main service
main_log.txt
There is not much difference between the two parameters, the difference is that the main service outputs a prompt when loading vocab
The text was updated successfully, but these errors were encountered: