👋 join us on Discord and WeChat

If you like Dingo, please give us a ⭐ on GitHub!

Introduction

Dingo is A Comprehensive AI Data, Model and Application Quality Evaluation Tool, designed for ML practitioners, data engineers, and AI researchers. It helps you systematically assess and improve the quality of training data, fine-tuning datasets, and production AI systems.

---

🚀 Enterprise Dingo SaaS Version

Need a production-grade data quality platform? Try Dingo SaaS Enterprise Edition!

✨ Compared to the open-source version, SaaS provides:

• 🌐 Web UI - Visual evaluation interface, no coding required
• 🔐 Access Control - JWT + Google OAuth 2.0
• 📊 Visual Reports - Interactive charts, trend analysis, export features
• 🔌 RESTful API - Seamless integration with existing systems

📝 How to Get Free SaaS Code

👉 Apply for Dingo SaaS Repository Access

Review time: 1-5 business days | Suitable for enterprise data governance, team collaboration

---

Why Dingo?

🎯 Production-Grade Quality Checks - From pre-training datasets to RAG systems, ensure your AI gets high-quality data

🗄️ Multi-Source Data Integration - Seamlessly connect to Local files, SQL databases (PostgreSQL/MySQL/SQLite), HuggingFace datasets, and S3 storage

🔍 Multi-Field Evaluation - Apply different quality rules to different fields in parallel (e.g., ISBN validation for isbn, text quality for title)

🤖 RAG System Assessment - Comprehensive evaluation of retrieval and generation quality with 5 academic-backed metrics

🧠 LLM & Rule & Agent Hybrid - Combine fast heuristic rules (30+ built-in) with LLM-based deep assessment

🚀 Flexible Execution - Run locally for rapid iteration or scale with Spark for billion-scale datasets

📊 Rich Reporting - Detailed quality reports with GUI visualization and field-level insights

Architecture Diagram

Quick Start

Installation

# Core package (includes rule evaluation, LLM evaluation, MCP server, datasource support)
pip install dingo-python

# With HHEM hallucination detection model (requires transformers + torch)
pip install "dingo-python[hhem]"

# With all features (HHEM + Agent)
pip install "dingo-python[all]"

Example Use Cases of Dingo

1. Evaluate LLM chat data

from dingo.config.input_args import EvaluatorLLMArgs
from dingo.io.input import Data
from dingo.model.llm.text_quality.llm_text_quality_v4 import LLMTextQualityV4
from dingo.model.rule.rule_common import RuleSpecialCharacter

data = Data(
    data_id='123',
    prompt="hello, introduce the world",
    content="�I am 8 years old. ^I love apple because:"
)


def llm():
    LLMTextQualityV4.dynamic_config = EvaluatorLLMArgs(
        key='YOUR_API_KEY',
        api_url='https://api.openai.com/v1/chat/completions',
        model='gpt-4o',
    )
    res = LLMTextQualityV4.eval(data)
    print(res)

def rule():
    res = RuleSpecialCharacter().eval(data)
    print(res)

rule()

2. Evaluate Dataset

from dingo.config import InputArgs
from dingo.exec import Executor

# Evaluate a dataset from Hugging Face
if __name__ == '__main__':
    input_data = {
        "input_path": "tatsu-lab/alpaca",  # Dataset from Hugging Face
        "dataset": {
            "source": "hugging_face",
            "format": "plaintext"  # Format: plaintext
        },
        "executor": {
            "result_save": {
                "bad": True  # Save evaluation results
            }
        },
        "evaluator": [
            {
                "evals": [
                    {"name": "RuleColonEnd"},
                    {"name": "RuleSpecialCharacter"}
                ]
            }
        ]
    }

    input_args = InputArgs(**input_data)
    executor = Executor.exec_map["local"](input_args)
    result = executor.execute()
    print(result)

Command Line Interface

Evaluate with Rule Sets

dingo eval --input .github/env/local_plaintext.json

Evaluate with LLM (e.g., GPT-4o)

dingo eval --input .github/env/local_json.json

---

MCP Server

Dingo includes a built-in Model Context Protocol (MCP) server for AI agent integration:

# Start MCP server (SSE transport, default port 8000)
dingo serve

# Custom port
dingo serve --port 9000

# stdio transport (for Claude Desktop)
dingo serve --transport stdio

For detailed setup and Cursor/Claude Desktop integration, see the dedicated documentation:

English · 简体中文 · 日本語