Tesla T4 got error

[xusysh]

Hi there, I'm using the image that you guys provided to test codegeex on Tesla T4 GPUs, the DSM is set to 75
after cmake build and checkpoint convertion, I ran api.py with datatype=int8, and the model successfully loaded

I used input() to make a breakpoint here, after I continue the program, the model predict got "runtime error", seems like it's from the C++ source code

[JadeLuo]

The ultimate reason for this error is that your GPU memory capacity is insufficient. You can try to lower the precision first by changing the default parameter of --data_type in the api.py file to int8, that is, parser.add_argument(‘–data_type’, type=str, choices=[‘fp16’, ‘int8’], default=‘int8’). The underlying code that reports Runtime error can be found in /workspace/codegeex-fastertransformer/src/fastertransformer/th_op/codegeex/CodegeexOp.h: remove the exception handling of codegeex.forward(&output_tensors, &input_tensors, &codegeex_weights_); and you will see a prompt that cuda memory is insufficient.报这个错误的最终原因是你的GPU显存容量不足，你可以先尝试降低精度，个改api.py文件中的--data_type默认参数为int8,即parser.add_argument('--data_type', type=str, choices=['fp16', 'int8'], default='int8')，报Runtime error的底层代码可以找到/workspace/codegeex-fastertransformer/src/fastertransformer/th_op/codegeex/CodegeexOp.h：取消掉codegeex.forward(&output_tensors, &input_tensors, &codegeex_weights_);异常捕获，即能看到提示cuda显存不足。

[Micro0110]

same question

[tutuxxx]

same question