# Orchestration

## Overview
The Orchestration layer coordinates multi-step workflows between Connectors, the RAG Service, and the Enterprise Brain Agent. It manages fan-out/fan-in tasks, retries, backoffs, and idempotency for reliable execution. Additionally, it enables schema-aware query orchestration to ensure intelligent and context-driven responses.

## Responsibilities
- Sequence workflows: retrieve → augment → generate → persist.
- Coordinate cross-connector actions when needed.
- Manage background jobs and scheduled executions.
- Handle transient failures with retries and circuit breakers.
- Enable schema-aware query planning and execution.
- Ensure traceability and reproducibility of workflows.

## Technical Specifications

### Workflow Engine
- Support synchronous and asynchronous flows.
- Idempotent operations with unique workflow IDs.
- Dead-letter queue for failed jobs.
- Deterministic execution pipelines for schema-aware queries.

### Schema-Aware Query Orchestration
- **Schema Parsing**: Load and parse the application schema (`app-schema.md`) at startup. Convert it into a structured JSON format for agent consumption.
- **Semantic Registry**: Build a semantic schema registry enriched with business semantics (e.g., table purposes, searchable fields, vector fields, timestamp fields).
- **Query Strategy Selection**: Classify user queries into execution strategies (e.g., `sql_only`, `vector_only`, `sql_then_rag`).
- **Execution Flow**: Orchestrate hybrid query execution in the following order:
  - SQL filtering
  - Vector ranking (if required)
  - RAG interpretation (if required)
- **Validation**: Validate all queries against the schema registry to ensure safety and correctness.

### Reliability
- Exponential backoff and jitter for retries.
- Structured error propagation and alerting hooks.
- Validation layers to prevent invalid queries or workflows.

### Observability
- Emit traces, metrics, and logs for each workflow step.
- Per-workflow dashboards for health and SLA tracking.
- Structured logging for debugging and performance monitoring.

## Interfaces
- API to start workflows (e.g., `askWithRetrieval`, `bulkReindex`).
- Event hooks for connectors and RAG callbacks.
- Query validation and execution endpoints.

## Orchestration Agent Details

The Orchestration Agent is the core component responsible for planning, executing, and managing schema-aware queries. It ensures that workflows are deterministic, traceable, and aligned with the business context defined in the schema registry.

### Key Features
1. **Autonomous Query Planning**:
   - Dynamically classify user queries into execution strategies.
   - Select appropriate connectors and data sources based on the schema registry.

2. **Hybrid Query Execution**:
   - Combine SQL filtering, vector similarity, and RAG interpretation in a single pipeline.
   - Maintain intermediate results for traceability.

3. **Validation and Safety**:
   - Enforce schema constraints to prevent invalid queries.
   - Ensure all queries are read-only and adhere to governance policies.

4. **Observability and Debugging**:
   - Emit structured logs and metrics for each step of the workflow.
   - Provide detailed traces for debugging and performance analysis.

5. **Reliability**:
   - Handle transient failures with retries and backoffs.
   - Use dead-letter queues for failed jobs.

### Execution Pipeline
The Orchestration Agent follows a deterministic execution pipeline:
1. **Schema Parsing**: Load and validate the schema at startup.
2. **Query Strategy Selection**: Classify the user query and generate a structured execution plan.
3. **SQL Query Execution**: Filter data using schema-aware SQL queries.
4. **Vector Query Execution**: Perform vector similarity searches on SQL-filtered results.
5. **RAG Interpretation**: Use the RAG service to generate context-aware responses.
6. **Answer Composition**: Combine results and include supporting evidence.
7. **Logging and Metrics**: Emit logs and metrics for each step.

### Benefits
- **Intelligence**: Enables the agent to autonomously plan and execute complex queries.
- **Safety**: Ensures all queries are schema-aware and adhere to governance policies.
- **Traceability**: Provides detailed logs and metrics for debugging and monitoring.
- **Scalability**: Supports hybrid execution paths for large-scale data retrieval and processing.