Turn text input into actions with Needle, a 14MB function-calling LLM

Overview of Needle 2

Cactus Compute's Needle 2 is a lightweight, function-calling LLM designed for on-device execution on a Raspberry Pi 5. Unlike traditional chatbots, Needle 2 is trained for a specific purpose: executing structured actions based on natural language input. It operates entirely offline, without requiring a cloud API or network connection, making it suitable for environments where internet access is limited or undesirable.

Functionality and Use Cases

Needle 2 works by accepting Python functions decorated with the @needle.tool decorator. These functions are then used by the model to determine which action to take based on user input. For example, the model can be instructed to 'Turn the LED on,' and it will execute the corresponding function to control the LED on the Raspberry Pi 5.

The model's ability to execute local actions is particularly useful for applications such as home automation, embedded systems, and IoT devices. It can be used to write local notes, query system information, interact with hardware components like LEDs, and more. The model's performance is demonstrated through various examples, including saving notes, retrieving CPU temperature, and controlling hardware components.

Performance and Limitations

Needle 2 is designed to be efficient, with a native session size of around 28MB. In practice, the model's memory usage peaks between 43MB and 46.4MB, including the interpreter. The model's response times are generally fast, with examples showing execution times ranging from 78ms to 149ms for different tasks.

However, Needle 2 is not a general-purpose chatbot. It is limited to the functions explicitly defined and decorated with @needle.tool. If a user asks a question that does not match any of the available functions, the model will not provide an answer. For instance, asking 'What is the capital of France?' results in no function call being made, as the model does not have access to external knowledge or a broader set of tools.

Implementation and Customization

Implementing Needle 2 involves defining Python functions with clear names, type annotations, and descriptions. These functions are then passed to the Needle agent, which uses them to execute actions based on user input. The model can be fine-tuned locally on a laptop for a specific set of tools, and the weights and code are available on Hugging Face and GitHub under the Apache 2.0 license.

Needle 2 is particularly useful for developers who are already familiar with Python and want to integrate natural language processing into their existing codebase. It can be used to enhance user interfaces by allowing users to interact with devices using plain language instead of strict API calls.

Conclusion

Cactus Compute's Needle 2 is a promising tool for on-device execution of structured actions based on natural language input. Its lightweight design and offline capabilities make it suitable for a wide range of applications, particularly in embedded systems and IoT devices. However, its limitations in terms of general knowledge and reliance on predefined functions mean that it is best suited for specific use cases where the available tools are well-defined and limited in scope.

Source: Raspberry Pi. Analysis based on publicly available information; no hands-on testing is claimed.