# Promptfoo Testing Guide for Enterprise Brain

## Table of Contents

- [Overview](#overview)
- [Setup & Installation](#setup--installation)
- [YAML Configuration Structure](#yaml-configuration-structure)
- [Golden Answer Testing](#golden-answer-testing)
- [Running Tests](#running-tests)
- [Viewing Results](#viewing-results)
- [Troubleshooting](#troubleshooting)
- [Best Practices](#best-practices)

---

## Overview

**Promptfoo** is an LLM testing framework that validates agent responses against expected outputs. In Enterprise Brain, we use it to:

- ✅ Test Chanakya agent responses with **golden answers**
- ✅ Validate JSON structure and status codes
- ✅ Ensure key facts appear in agent responses
- ✅ Perform regression testing after agent updates
- ✅ Generate visual reports for quality assurance

---

## Setup & Installation

### Prerequisites

- **Node.js**: v20.x or v22.x
- **nvm** (Node Version Manager) - recommended
- **Chanakya Agent**: Running on `http://localhost:8010`

### Installation Steps

#### 1. Install Node.js (if not installed)

```bash
# Install nvm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash

# Install Node 22
nvm install 22
nvm use 22
```

#### 2. Install Promptfoo

We use **promptfoo 0.90.0** (compatible with Node 22):

```bash
npm install -g promptfoo@0.90.0
```

**⚠️ Important**: Do NOT use promptfoo 0.100.0+ with Node 22 due to ESM compatibility issues.

#### 3. Verify Installation

```bash
promptfoo --version
# Should output: 0.90.0

node --version
# Should output: v22.x.x
```

#### 4. Set Node Version for Project

Create `.nvmrc` in project root (already exists):

```bash
echo "22" > .nvmrc
```

Now `nvm use` will automatically switch to Node 22 when in the project directory.

---

## YAML Configuration Structure

Promptfoo uses YAML configuration files to define tests. Here's the complete structure:

### Basic Structure

```yaml
# Configuration metadata
description: <Test suite description>

# Prompt templates
prompts:
  - "{{variable_name}}"

# Provider configuration (API endpoint)
providers:
  - id: <endpoint_url>
    label: <provider_name>
    config:
      method: <HTTP_method>
      headers:
        <header_name>: <header_value>
      body:
        <param_name>: "{{variable}}"
      responseParser: json

# Test cases
tests:
  - vars:
      <variable_name>: <value>
      <golden_answer>: <expected_value>
    assert:
      - type: <assertion_type>
        value: <expected_value>

# Default test configuration
defaultTest:
  options:
    transformProvider: json
  assert:
    - type: <assertion_type>
```

### Enterprise Brain Example

Here's our actual configuration from `promptfooconfig-agents.yaml`:

```yaml
description: Chanakya Enterprise Brain - Golden Answer Validation

prompts:
  - "{{question}}"

providers:
  - id: http://localhost:8010/query
    label: Chanakya Hub
    config:
      method: POST
      headers:
        Content-Type: application/json
      body:
        question: "{{prompt}}"
        user_email: "test@divami.com"
        conversation_id: null
      responseParser: json

tests:
  # Test 1: Simple golden answer validation
  - vars:
      question: "What is the Base Contract Value for Tenova?"
      golden_answer: "£99,832,916"
    assert:
      - type: is-json
      - type: contains
        value: '"status":"success"'
      - type: icontains
        value: "99,832,916"

  # Test 2: Name matching
  - vars:
      question: "Who is the Contract Manager for ABB?"
      golden_answer: "Samad Mohammad"
    assert:
      - type: is-json
      - type: contains
        value: '"status":"success"'
      - type: icontains
        value: "Samad Mohammad"

defaultTest:
  options:
    transformProvider: json
  assert:
    - type: is-json
    - type: contains
      value: '"status"'
```

---

## Golden Answer Testing

### What is Golden Answer Testing?

**Golden answers** are pre-validated correct responses used as a baseline. Tests compare agent output against these golden answers to ensure consistency and accuracy.

### Golden Answer Structure

Each test includes:

1. **`question`** - The prompt/query to test
2. **`golden_answer`** - The expected correct answer (for documentation)
3. **Assertions** - Validation rules to check key facts

### Example Test Block

```yaml
- vars:
    # The question to ask the agent
    question: "What is the Total Commitment (%) for Churngold?"

    # Expected answer (documented for reference)
    golden_answer: "173%"

  assert:
    # Response must be valid JSON
    - type: is-json

    # Response must have success status
    - type: contains
      value: '"status":"success"'

    # Response must contain the key fact (case-insensitive)
    - type: icontains
      value: "173"
```

### Assertion Types

#### 1. `is-json`

Validates that the response is valid JSON.

```yaml
- type: is-json
```

#### 2. `contains` (Case-Sensitive)

Checks if response contains exact string.

```yaml
- type: contains
  value: '"status":"success"'
```

#### 3. `icontains` (Case-Insensitive)

Checks if response contains string (ignores case).

```yaml
- type: icontains
  value: "Samad Mohammad"
```

#### 4. `javascript` (Custom Logic)

Execute custom JavaScript for complex validation.

```yaml
- type: javascript
  value: |
    output.status === 'success' ? true : false
```

**⚠️ Note**: JavaScript assertions in promptfoo 0.90.0 must return explicit `true`/`false` booleans, not truthy values.

---

## Running Tests

### Basic Commands

#### 1. Run Tests

```bash
promptfoo eval -c promptfooconfig-agents.yaml
```

This executes all tests in the configuration file and displays results in the terminal.

#### 2. View Web UI

```bash
promptfoo view
```

Opens a web interface at `http://localhost:15500` to explore test results visually.

#### 3. Run Both (Eval + View)

```bash
promptfoo eval -c promptfooconfig-agents.yaml && promptfoo view
```

### Advanced Options

#### Run Specific Number of Tests

```bash
promptfoo eval -c promptfooconfig-agents.yaml --max-concurrency 5
```

#### Filter Tests

```bash
promptfoo eval -c promptfooconfig-agents.yaml --filter-description "Tenova"
```

#### Generate Report

```bash
promptfoo eval -c promptfooconfig-agents.yaml --output results.json
```

#### Share Results

```bash
promptfoo share
```

Creates a shareable URL for test results.

---

## Viewing Results

### Terminal Output

After running `promptfoo eval`, you'll see:

```
┌───────────────────────┬───────────────────────┐
│ question              │ [Chanakya Hub]        │
├───────────────────────┼───────────────────────┤
│ What is the Base      │ [PASS] ✓              │
│ Contract Value for    │                       │
│ Tenova?               │                       │
├───────────────────────┼───────────────────────┤
│ Who is the Contract   │ [PASS] ✓              │
│ Manager for ABB?      │                       │
└───────────────────────┴───────────────────────┘

✔ Evaluation complete.
```

### Web UI

The web interface (`promptfoo view`) shows:

- **Summary Dashboard**: Pass/fail rates, average scores
- **Test Details**: Individual test results with full responses
- **Response Viewer**: Raw JSON responses from the agent
- **Assertion Details**: Which assertions passed/failed
- **Comparison View**: Compare multiple test runs

#### Web UI Features

1. **Filter by Status**: Show only failed tests
2. **Search**: Find specific questions or answers
3. **Export**: Download results as JSON/CSV
4. **History**: View previous test runs

---

## Troubleshooting

### Common Issues

#### 1. `Error: No configuration file found`

**Solution**: Ensure you're in the correct directory and the YAML file exists.

```bash
ls promptfooconfig-agents.yaml  # Should list the file
pwd  # Should be: /path/to/ai-enterprise-brain
```

#### 2. `Connection Refused` on `localhost:8010`

**Solution**: Start the Chanakya agent.

```bash
cd backend
make run-chanakya
```

Verify it's running:

```bash
curl http://localhost:8010/health
```

#### 3. Tests Failing with `Custom function returned false`

**Solution**: JavaScript assertions must return explicit `true`/`false`.

❌ **Wrong**:

```yaml
- type: javascript
  value: output.status === 'success' # Returns string 'success'
```

✅ **Correct**:

```yaml
- type: javascript
  value: |
    output.status === 'success' ? true : false
```

#### 4. YAML Parsing Errors

**Solution**: Check indentation and syntax.

- Use **2 spaces** for indentation (not tabs)
- Ensure proper nesting
- Quote strings with special characters

```yaml
# ❌ Wrong
question: What is the "Quote" value?

# ✅ Correct
question: 'What is the "Quote" value?'
```

#### 5. `promptfoo view` Shows No Results

**Solution**: Run `promptfoo eval` first to generate results.

```bash
promptfoo eval -c promptfooconfig-agents.yaml
promptfoo view
```

#### 6. Version Compatibility Issues

**Solution**: Use Node 22 + promptfoo 0.90.0.

```bash
nvm use 22
npm uninstall -g promptfoo
npm install -g promptfoo@0.90.0
```

---

## Best Practices

### 1. Golden Answer Maintenance

✅ **Do**:

- Update golden answers when business requirements change
- Verify golden answers against source of truth (database, stakeholders)
- Document why specific values are expected

❌ **Don't**:

- Use outdated or unverified golden answers
- Copy agent responses blindly as golden answers
- Mix test data with production data

### 2. Assertion Strategy

✅ **Do**:

- Use `icontains` for case-insensitive matching (names, entities)
- Check for key numeric values (contract amounts, percentages)
- Validate JSON structure and status codes
- Layer assertions from general to specific

❌ **Don't**:

- Use exact string matching for generated text
- Over-specify assertions (too many keywords)
- Ignore error handling tests

### 3. Test Organization

```yaml
tests:
  # Group 1: Basic Data Retrieval
  - vars:
      question: "What is X?"
    # ...

  # Group 2: Complex Queries
  - vars:
      question: "Compare X and Y"
    # ...

  # Group 3: Error Handling
  - vars:
      question: "Nonexistent entity xyz123"
    # ...
```

### 4. Naming Conventions

- Use descriptive variable names: `question`, `golden_answer`
- Comment test blocks with test numbers and descriptions
- Keep YAML file names descriptive: `promptfooconfig-agents.yaml`

### 5. Test Coverage

Ensure tests cover:

- ✅ Simple fact retrieval
- ✅ Complex aggregations
- ✅ Date/time queries
- ✅ Multi-entity comparisons
- ✅ Error scenarios
- ✅ Edge cases

---

## Example: Adding a New Test

### Step 1: Identify Question and Golden Answer

**Question**: "What is the total value of Bath Demolition contracts?"  
**Golden Answer**: "£6,543,210"

### Step 2: Add Test Block

```yaml
tests:
  # ... existing tests ...

  # Test 31: Bath Demolition Total Value
  - vars:
      question: "What is the total value of Bath Demolition contracts?"
      golden_answer: "£6,543,210"
    assert:
      - type: is-json
      - type: contains
        value: '"status":"success"'
      - type: icontains
        value: "6,543,210"
      - type: icontains
        value: "Bath Demolition"
```

### Step 3: Run Test

```bash
promptfoo eval -c promptfooconfig-agents.yaml
```

### Step 4: Review Results

```bash
promptfoo view
```

Check if the test passes. If not, verify:

- Agent is returning correct data
- Golden answer matches database
- Assertions are checking for the right values


---

## Configuration Files

### Main Configuration

**File**: `promptfooconfig-agents.yaml`  
**Purpose**: Golden answer testing for Chanakya agent  
**Tests**: 30 business query validations  
**Endpoint**: `http://localhost:8010/query`

### Alternative Configurations

You can create multiple configuration files for different test suites:

- `promptfooconfig-regression.yaml` - Regression tests
- `promptfooconfig-performance.yaml` - Performance benchmarks
- `promptfooconfig-edge-cases.yaml` - Edge case validation

Run each with:

```bash
promptfoo eval -c promptfooconfig-<name>.yaml
```

---

## Useful Commands Reference

```bash
# Installation
npm install -g promptfoo@0.90.0

# Run tests
promptfoo eval -c promptfooconfig-agents.yaml

# View results in browser
promptfoo view

# Run and view
promptfoo eval -c promptfooconfig-agents.yaml && promptfoo view

# Generate JSON report
promptfoo eval -c promptfooconfig-agents.yaml --output results.json

# Share results
promptfoo share

# Check version
promptfoo --version

# Get help
promptfoo --help
promptfoo eval --help
```

---

## Resources

- **Promptfoo Documentation**: https://promptfoo.dev/docs
- **GitHub Repository**: https://github.com/promptfoo/promptfoo


---

## Next Steps

1. **Run Current Tests**:

   ```bash
   promptfoo eval -c promptfooconfig-agents.yaml
   promptfoo view
   ```

2. **Add More Tests**:
   - Identify additional business queries
   - Define golden answers
   - Add test blocks to YAML

3. **Expand Coverage**:
   - Add edge case tests
   - Test error handling
   - Add performance benchmarks

---

**Last Updated**: April 7, 2026  
**Maintainer**: Enterprise Brain Team