You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I updated the LLM Module's firmware to v1.3.0 with AXDL , and installed llm-openbuddy-llama3.2-1B-ax630c_0.2-m5stack1_arm64.deb via SD card. But I get only null reply with the latest version of M5Module-LLM library, when I specify the model name of "openbuddy-llama3.2-1B-ax630c" to setup method.
Is there any other instruction needed to use the openbuddy-llama3.2 model?
It seems that the model has been installed into ModuleLLM. The problem is in the loading model. You can try to use the following json to load the model to determine whether it is successfully loaded.
{
"request_id": "llm001",
"work_id": "llm",
"action": "setup",
"object": "llm.setup",
"data": {
"model": "openbuddy-llama3.2-1B-ax630c",
"response_format": "llm.utf-8.stream",
"input": "llm.utf-8",
"enoutput": true,
"max_token_len": 127,
"prompt": "You are a knowledgeable assistant capable of answering various questions and providing information."
}
}
Ok, I got it work. Thank you Abandon-ht !
The problem was that it timeouts during model setup.
The openbuddy-llama3.2-1B-ax630c model takes about 15 sec to setup but the timeout value for the setup was 10sec on M5Module-LLM. So it timeouts and the returned work_id was null. Then the inference did not work.
Activity
Abandon-ht commentedon Dec 30, 2024
Is the platform you use Arduino or UIFlow?
Abandon-ht commentedon Dec 30, 2024
https://uiflow2.m5stack.com/?pkey=57b3b457cf2247ce827a09d3cc245e5d You can refer to this method to make sure that you have successfully followed this model.
mhama commentedon Dec 31, 2024
I'm using Arduino with the latest github version of this library https://github.com/m5stack/M5Module-LLM .
I searched for
get model listapi but I can't find it.In this repository there's
lsmodeapi. Maybe the UIFlow'sget model listapi calls this?StackFlow/projects/llm_framework/main_sys/src/event_loop.cpp
Line 194 in 9d1dfce
So I did a little hack to M5Module-LLM and send the command of "sys_lsmode".
I found this info about openbuddy model in the result:
{ "capabilities": [ "text_generation", "chat" ], "input_type": [ "llm.utf-8", "llm.utf-8.stream", "llm.chat_completion", "llm.chat_completion.stream" ], "mode": "openbuddy-llama3.2-1B-ax630c", "mode_param": { "axmodel_num": 16, "b_bos": false, "b_dynamic_load_axmodel_layer": false, "b_eos": false, "b_use_mmap_load_embed": true, "b_use_topk": false, "filename_post_axmodel": "llama_post.axmodel", "filename_tokenizer_model": "http://localhost:8080", "filename_tokens_embed": "model.embed_tokens.weight.bfloat16.bin", "template_filename_axmodel": "llama_p128_l%d_together.axmodel", "tokenizer_type": 2, "tokens_embed_num": 128256, "tokens_embed_size": 2048 }, "output_type": [ "llm.utf-8", "llm.utf-8.stream" ], "type": "llm" },This is just the same as this file:
https://github.com/m5stack/StackFlow/blob/9d1dfced01e365694f6aa69b78aed29d0bf49e26/projects/llm_framework/main_llm/mode_openbuddy-llama3.2-1B-ax630c.json
I'm using llm inference like this on .ino but it always replies with
null.llm_config = m5_module_llm::ApiLlmSetupConfig_t(); llm_config.model = "openbuddy-llama3.2-1B-ax630c"; llm_work_id = module_llm.llm.setup(llm_config); ... module_llm.llm.inferenceAndWaitResult(llm_work_id, question.c_str(), [](String& result) { /* Show result on screen */ M5.Display.printf("%s", result.c_str()); Serial.printf("%s", result.c_str()); });mhama commentedon Dec 31, 2024
I modified M5Module-LLM library and output error code/message at inference (ApiLlm::inferenceAndWaitResult).
So maybe it fails around here:
StackFlow/projects/llm_framework/main_sys/src/event_loop.cpp
Line 598 in 9d1dfce
Abandon-ht commentedon Jan 2, 2025
It seems that the model has been installed into ModuleLLM. The problem is in the loading model. You can try to use the following json to load the model to determine whether it is successfully loaded.
{ "request_id": "llm001", "work_id": "llm", "action": "setup", "object": "llm.setup", "data": { "model": "openbuddy-llama3.2-1B-ax630c", "response_format": "llm.utf-8.stream", "input": "llm.utf-8", "enoutput": true, "max_token_len": 127, "prompt": "You are a knowledgeable assistant capable of answering various questions and providing information." } }mhama commentedon Jan 2, 2025
Ok, I got it work. Thank you Abandon-ht !
The problem was that it timeouts during model setup.
The
openbuddy-llama3.2-1B-ax630cmodel takes about 15 sec to setup but the timeout value for the setup was 10sec on M5Module-LLM. So it timeouts and the returned work_id was null. Then the inference did not work.Merge pull request #2 from yuyun2000/opt/melotts