python web_demo.py Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Traceback (most recent call last): File "/home/nano/THUDM/ChatGLM-6B/web_demo.py", line 5, in <module> tokenizer = AutoTokenizer.from_pretrained("/home/nano/THUDM/chatglm-6b", trust_remote_code=True) File "/home/nano/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 679, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) File "/home/nano/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1804, in from_pretrained return cls._from_pretrained( File "/home/nano/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1958, in _from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) File "/home/nano/.cache/huggingface/modules/transformers_modules/chatglm-6b/tokenization_chatglm.py", line 221, in __init__ self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens) File "/home/nano/.cache/huggingface/modules/transformers_modules/chatglm-6b/tokenization_chatglm.py", line 64, in __init__ self.text_tokenizer = TextTokenizer(vocab_file) File "/home/nano/.cache/huggingface/modules/transformers_modules/chatglm-6b/tokenization_chatglm.py", line 22, in __init__ self.sp.Load(model_path) File "/home/nano/.local/lib/python3.10/site-packages/sentencepiece/__init__.py", line 905, in Load return self.LoadFromFile(model_file) File "/home/nano/.local/lib/python3.10/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg) RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]什么错误
时间: 2023-12-08 08:04:59 浏览: 425
在Windows服务器下用Apache和mod_wsgi配置Python应用的教程
5星 · 资源好评率100%
根据您提供的错误信息,看起来是加载模型时出现了一些问题。具体来说,这个错误是由于加载了一个具有自定义代码的模型,并且在加载时没有明确指定修订版本。为了确保没有恶意代码被贡献到较新的修订版本中,建议显式地传递一个`revision`参数。
您可以尝试以下解决方法:
在`web_demo.py`文件中,将以下代码行:
```python
tokenizer = AutoTokenizer.from_pretrained("/home/nano/THUDM/chatglm-6b", trust_remote_code=True)
```
替换为:
```python
tokenizer = AutoTokenizer.from_pretrained("/home/nano/THUDM/chatglm-6b", trust_remote_code=True, revision="main")
```
这样会明确指定修订版本为"main",以避免加载模型时出现安全问题。请确保您已经下载了正确的模型文件并将路径指向正确的位置。
如果问题仍然存在,请提供更多详细信息,以便我能够更好地帮助您解决问题。
阅读全文