perturber: A Stateless Service for Semantics-Preserving Text Perturbations
The perturber tool introduces a stateless service and Python library designed to apply single, semantics-preserving text perturbations. It offers a catalog of 68 perturbations across 11 families, enabling systematic evaluation of model robustness by isolating the impact of individual perturbations.
Overview of perturber
The perturber tool is a stateless service and Python library that applies exactly one semantics-preserving text perturbation per call. It is designed to address a critical gap in the evaluation of large language models (LLMs), where clean benchmarks may overstate model robustness. Real-world inputs often arrive perturbed, and these variations can significantly affect model outputs. Existing tools often couple perturbations to a benchmarking stack, making them less reusable as standalone transforms.
Key Features and Design
perturber introduces a first-party catalog of 68 perturbations across 11 families, enabling users to apply individual perturbations systematically. The service and library are designed to produce identical output by construction, ensuring consistency across different environments. This is achieved by relying on a single primitive operation, with composition left to the caller through chaining of calls.
The tool focuses on evaluating its own correctness, reproducibility, and cost, rather than the robustness of any specific model. It measures meaning-preservation per perturbation, avoiding assumptions about the impact of perturbations on model behavior. This approach allows for a more accurate and systematic study of how perturbations affect model outputs.
Use Cases and Evaluation
perturber includes a live playground, an introspectable catalog for benchmark selection, and a one-command reproducible evaluation. These features make it easier for researchers and developers to test and evaluate the impact of different perturbations on model behavior. The tool is particularly useful in scenarios where systematic evaluation of model robustness is required, such as in benchmarking and testing environments.
The demonstration of perturber highlights its ability to apply perturbations in a reproducible and consistent manner, making it a valuable tool for evaluating the robustness of LLMs. By isolating the impact of individual perturbations, perturber enables a more accurate understanding of how real-world inputs affect model outputs.
Source: arXiv NLP. Analysis based on publicly available information; no hands-on testing is claimed.
