Show HN: LLMxRay an open-source observability tool for LLMs

Overview of LLMxRay

LLMxRay is an open-source observability tool tailored for developers, researchers, and educators working with local large language models (LLMs). It provides a comprehensive interface for monitoring and analyzing the behavior of models running on Ollama, a local LLM server. The tool is designed to help users understand how their AI models are performing in real time, without the need for cloud services or API keys.

Key Features

LLMxRay offers a range of features aimed at improving the transparency and efficiency of working with local LLMs. These include:

• Real-Time Token Streaming: Users can observe how tokens are generated in real time, with each token colored based on the speed of its generation. This helps identify where the model is hesitating or uncertain.
• Quality Analysis: The tool automatically analyzes responses and highlights potential issues such as repetition, refusal, gibberish, or truncation. Colored badges appear when these issues are detected, providing immediate visual feedback.
• Model Comparison: LLMxRay allows users to compare up to four different models, temperatures, or system prompts side by side. This feature includes metrics comparison, word-level diff highlighting, and one-click presets for common scenarios.
• Performance Analytics: The tool provides detailed performance metrics, including latency percentiles, error intelligence, and usage heatmaps. It also tracks the impact of different settings on model performance.
• Cost Dashboard: Users can track token usage per model and see estimated cloud-equivalent pricing. This helps quantify the cost savings of running models locally.
• Surgical Benchmark: This feature allows users to test model knowledge with multi-choice question suites, using real logprobs for accurate confidence measurement. Custom benchmarks can be created visually or generated by AI.
• Embeddings Lab & RAG Pipeline: Users can embed text, visualize vectors, and measure cosine similarity. The tool also supports building a local knowledge base from PDFs, DOCX, and CSV files.
• Tool Workshop: A visual canvas for building tool definitions, with bidirectional code sync between nodes and TypeScript. This allows for the creation and testing of custom tools.
• Protocol Observatory: This feature allows users to fire the same prompt through Ollama's three serving protocols in parallel, providing side-by-side streaming and per-protocol metrics.

Use Cases and Target Audience

LLMxRay is particularly useful for developers who need to debug prompts, profile latency, and compare models. Researchers can use it to run controlled experiments with consistent settings across models and temperatures. Educators and students can benefit from the built-in Educators Kit, which includes nine interactive modules for exploring model behavior visually. AI team leads can use the tool to understand quality trends, error patterns, and resource usage across their local fleet of models.

Installation and Compatibility

LLMxRay can be installed using a single command with npx or via Docker. It is compatible with Ollama 0.20 and newer, with recommended versions being 0.33.3+ for full feature support. The tool uses several Ollama endpoints for streaming chat, generation, model listing, and embeddings, ensuring seamless integration with local LLM environments.

Conclusion

LLMxRay is a powerful open-source tool that enhances the observability and efficiency of working with local large language models. Its comprehensive feature set, including real-time token streaming, quality analysis, and model comparison, makes it a valuable asset for developers, researchers, and educators. By providing detailed insights into model behavior and performance, LLMxRay helps users make informed decisions about their AI workflows.

Source: Hacker News Show. Analysis based on publicly available information; no hands-on testing is claimed.