🧠TruthTorchLM: A Comprehensive Library for Truthfulness Assessment in LLMs
Date:
🔥 Introducing TruthTorchLM – an open-source Python library designed to assess and predict the truthfulness of LLM-generated outputs across both short-form and long-form content. Developed over the course of a year, TruthTorchLM brings together 30+ cutting-edge methods proposed in recent literature for truthfulness assessment and uncertainty quantification.
🚀 What TruthTorchLM Offers
- 30+ Truth Methods: Includes methods like Google search check, uncertainty-based scores, self-detection, and multi-LLM collaboration.
- Seamless Integration: Easily integrates with Huggingface and LiteLLM with minimal code changes.
- Evaluation Tools: Built-in functions to benchmark methods using AUROC, AUPRC, PRR, and Accuracy.
- Calibration Functions: Normalize output scores for meaningful comparison across methods.
- Long-Form Truthfulness: Automatically decomposes long responses into factual claims and assesses their correctness.
- Extendability: Easily plug in your own truth methods or evaluation pipelines.