> ## Content Index > Fetch the complete content index at: https://www.stirup.si/llms.txt > Use this file to discover other available public pages before exploring further. # Model Size and Precision Trade-offs in Browser-Based RAG - URL: https://www.stirup.si/model-size-and-precision-trade-offs-in-browser-based-rag-4b9d2298/ - Published: 2026-10-13T17:59:04.000Z - Updated: 2026-10-13T17:59:04.000Z - Description: Running RAG with small language models in the browser introduces trade-offs between model size and precision, as 4-bit builds can be larger than 4-bit-with-16-bit-floats versions due to full 32-bit precision in embedding tables. - Author: Satish Das - Tags: Community, News, Stirup Research ## Model Size and Precision Trade-offs Running RAG with a small language model in the browser introduces trade-offs between model size and precision. According to [Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si), models with 25 to 360 million parameters can be executed on users’ GPUs or CPUs, enabling local inference without relying on cloud services. However, [Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si) highlights an unexpected detail: the “4-bit” (q4) build of some models is larger than the 4-bit-with-16-bit-floats (q4f16) version. This occurs because the q4 export retains the embedding table in full 32-bit precision, increasing the file size. For example, SmolLM2-360M is 390 MB in the q4 format and 275 MB in the q4f16 format, with identical output, as noted in [Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si). These findings suggest that precision settings significantly affect model size, which can influence deployment and performance in browser environments. ## Model Performance on Specific Questions Model performance on specific questions can vary significantly, particularly when dealing with niche or highly technical inquiries. For example, when asked, “Which GPU should I buy to run Llama 70B locally?”, a model may generate responses that are either inaccurate or nonsensical. One instance cited in the evidence suggests the model answered with a string of repeated digits, such as “I'5555555555555555555555...”, indicating a failure to provide a coherent answer. This highlights the challenges of using smaller language models in the browser for complex tasks, where the lack of sufficient training data or context can lead to unreliable outputs. While some models may provide plausible-sounding but incorrect information, others may fail entirely, as noted in the blog’s discussion of similar issues. These examples underscore the importance of carefully evaluating model outputs, especially when making decisions based on their responses. ## Evaluation of Model Responses In the evaluation of model responses, all models were presented with the same four passages and identical prompts, with Opus 5.5 grading each answer as faithful, unsupported, or a non-answer based on the provided text ([Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si)). The analysis revealed that smaller models, such as Gemma 4 and Haiku, frequently responded with the statement “the passages don’t cover this,” indicating a tendency to avoid making unsupported claims ([Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si)). This behavior highlights a key advantage of smaller models in this context: they do not attempt to infer or fabricate information beyond the given passages. As noted, a 300 MB model does not need to know everything; it simply needs to read four passages and either summarise them or state that they do not answer the question ([Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si)). ## Mitigating Repetition in Model Outputs Mitigating repetition in model outputs is a critical step in refining the performance of language models, especially when running them in the browser with a small language model. One effective approach is the use of a repetition penalty, as noted in [Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si), where a value of 1.15 is applied to slightly discourage tokens the model has already seen. This technique helps in producing more coherent and varied outputs. However, it is important to recognize that such adjustments are not a universal solution and may require fine-tuning based on specific use cases. As highlighted in [Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si), the model's gate only catches obviously off-topic questions, leaving the rest to the model's internal mechanisms. While these strategies can improve output quality, they do not eliminate the need for further research and development in this area. ## Impact of Model Size on Faithfulness The impact of model size on faithfulness in RAG (Retrieval-Augmented Generation) systems was explored using Qwen3-0.6B and Gemma 4 E2B. In an experiment allowing models to use two, four, or six sentences, both models showed a decline in faithfulness as the number of sentences increased. However, the smaller Qwen3-0.6B model experienced a more significant drop, with faithful answers decreasing from 15 to 9 to 5 out of 30 as the number of sentences allowed increased. This suggests that while larger models may maintain a higher level of faithfulness even with more input, smaller models are more sensitive to changes in context length. The results highlight the trade-offs between model size and faithfulness in RAG systems, particularly when deploying these models in the browser with limited computational resources. ## Sources and reporting notes - [Lobsters](https://bart.degoe.de/rag-with-a-small-language-model-in-the-browser/?ref=stirup.si) Based on published source material. No independent hands-on testing is claimed.