> ## Content Index
> Fetch the complete content index at: https://www.stirup.si/llms.txt
> Use this file to discover other available public pages before exploring further.

# Performance Comparison of LLMs and GPUs with Optimization Tools
- URL: https://www.stirup.si/performance-comparison-of-llms-and-gpus-with-optimization-tools-1c77760f/
- Published: 2026-11-21T02:24:19.000Z
- Updated: 2026-11-21T02:24:19.000Z
- Description: A November 2026 study found significant variations in output tokens per second (tok/s) across large language models (LLMs), GPUs, and optimization tools, with expert settings outperforming stock configurations on specific hardware.
- Author: Satish Das
- Tags: Community, News, Stirup Research

## Performance Comparison Across Models and GPUs

In November 2026, a performance comparison was conducted across multiple large language models (LLMs) and GPUs, testing tools such as stock vLLM, SGLang, vLLM's auto\_tune, BentoML's llm-optimizer, and recommended engine settings. The results showed significant variations in output tokens per second (tok/s) depending on the model, GPU, and optimization tool used. For instance, on the Qwen3.5-35B-A3B (MoE) model with an H200 141 GB GPU, the best expert setting achieved 934 tok/s, outperforming the stock vLLM by −2.0 points. Similarly, on the Qwen3.5-9B model with an A100 SXM 80 GB GPU, the best expert setting achieved 581 tok/s, outperforming the stock vLLM by −0.9 points. These results highlight the importance of selecting the right configuration and optimization tool for specific models and hardware setups.

## Calibration and Serving Time

Calibration and serving time for large language models (LLMs) can vary significantly based on workload and hardware. According to one study, it takes 22–56 minutes per model and workload, with the time returning after 0.5–2.6 hours of busy serving **(**[**Hacker News Show**](https://github.com/Aagam-Bothara/rigtune?ref=stirup.si)**)**. When comparing methods, random sampling matched rigtune's staged search on all three 8B cards given the same time **(**[**Hacker News Show**](https://github.com/Aagam-Bothara/rigtune?ref=stirup.si)**)**. However, on a 14B model in fp8, the staged search outperformed random sampling by 4.4% **(**[**Hacker News Show**](https://github.com/Aagam-Bothara/rigtune?ref=stirup.si)**)**. These findings highlight the importance of efficient calibration strategies for optimizing LLM performance on GPUs. The results are specific to the tested configurations and do not imply universal guarantees for all models or hardware setups.

## Configuration Optimization and Tools

Configuration Optimization and Tools involve leveraging specialized frameworks to fine-tune Large Language Model (LLM) performance on specific hardware. Tools like vLLM's `auto_tune` and BentoML's `llm-optimizer` systematically adjust parameters such as batch cap and token budget to optimize under latency constraints, as noted in related studies. Early trials on an RTX 4090 showed SGLang leading before vLLM surpassed it through speculative decoding and quantized caching. However, some tools remain limited in scope, as one approach was described as 'very cheap' due to its lack of deployment, though it had no relevant data for the tested cards or model layers during these runs.

## Customization and Workload-Specific Tuning

Customization and workload-specific tuning are critical for optimizing large language model (LLM) performance on GPUs. According to user reports, configuring these settings can take 20–60 minutes per workload, but the results are tailored to specific prompts, offering precise outcomes for unique use cases. One user attempted to build a predictor for optimal configurations, but it only ranked the best-performing setup at a median of 15th out of 25 tested options, highlighting the limitations of predictive approaches in this domain. As such, prediction serves more as a filtering tool rather than a definitive decision-making mechanism. The developer encourages users to share their GPU details, model choices, and results in Discussions or to report issues, emphasizing the value of community feedback in refining the tool. This iterative process ensures that the solution evolves based on real-world usage and challenges.

## Installation and Hardware Requirements

To install and configure the system, first install the required packages using `pip install rigtune` and `pip install "vllm==0.33.0"`. The `vllm` version depends on the GPU driver: version 0.33.0 ran on driver 595, 0.29.0 requires 580+, and older drivers should use `vllm==0.11.0` with `transformers>=4.56,<5`. Once installed, launch the service with `rigtune serve Qwen/Qwen2.5-3B-Instruct --workload chat`. The first launch calibrates and caches results for the specific machine, model, objective, and workload. For custom tuning, pass a sample of your traffic using `--workload-file prompts.jsonl`, with one prompt per line. Optional features, such as suffix decoding, require additional steps, including building `arctic-inference` with specific CMake arguments to match the Python environment used by `vLLM`.

## Sources and reporting notes

- [Hacker News Show](https://github.com/Aagam-Bothara/rigtune?ref=stirup.si)

Based on published source material. No independent hands-on testing is claimed.