# Epic: E1 - Divami Enterprise Brain - Phase 1

- [ ] Epic Status
- Name: E1 - Divami Enterprise Brain - Phase 1
- Description: Deliver Phase-1 of the Enterprise Brain with connectors (GDrive, Gmail, GChat, Database, Web), a unified RAG Service, core platform features (Conversation & Feedback, Schedulers, User Management, Governance), and the Enterprise Brain Agent with Orchestration. This plan is the single source of execution truth and is intended for developers to follow directly.

## Direct Epic Tasks
- [ ] Database Design — Define schema, migrations, indexing strategy
- [ ] Project Setup — Backend (FastAPI), Frontend (Vite/React), CI (Jenkins)
- [ ] Gemini Keys Creation — Create and securely store LLM credentials

## Stories
### [ ]  Story S1: RAG Service - Core Retrieval Pipeline
   - Story Points: 8
   - Acceptance Criteria (Documentation):
      - A documented `ask` pipeline: query parsing → retrieval (BM25 + vector) → re-ranking → context assembly → generation.
      - Retrieval enforces RBAC/ACL filters by `user_id`, `space_id`, `acl`.
      - API returns answers with citations (source IDs) and metadata (chunks used, token counts, latency).
      - Basic observability: logs + metrics for retrieval and generation steps.
   - Related Docs: features/rag-service.md
#### Sub-Tasks
1. - [ ] Name: Define Chunk & Index Schemas
   - Description: Model chunk structure, metadata (source, owner, ACL, timestamps), and index selection (vector + BM25). Produce DB DDL and index settings.
   - Original Estimation: 8h
2. - [ ] Name: Implement Hybrid Retrieval
   - Description: Add BM25 keyword search and vector similarity with re-ranking. Include ACL filters and pagination.
   - Original Estimation: 12h
3. - [ ] Name: Context Window Assembler
   - Description: Assemble diverse, deduplicated context respecting token budget; include source attributions.
   - Original Estimation: 10h
4. - [ ] Name: Ask API and Response Schema
   - Description: Expose `/rag/ask` (or internal SDK) returning answer, citations, token usage, timings.
   - Original Estimation: 8h
5. - [ ] Name: Observability (Logs & Metrics)
   - Description: Emit structured logs and metrics for retrieval/generation latency, hit counts, token usage.
   - Original Estimation: 6h

### Story S2: GDrive Connector - Folder/File Sync
- [ ] Story Status
- Story Points: 8
- Acceptance Criteria (Documentation):
  - Authenticated sync retrieves folders and files with metadata (owner, size, timestamps, MIME type) and sharing ACLs.
  - Incremental sync using change tokens; idempotent operations; conflict-safe updates.
  - Attachments and file revisions tracked; errors retried with backoff; sync logged to `sync_logs`.
  - Data persists in `drive_folders` and `drive_files` tables with indexes.
- Related Docs: features/gdrive.md

#### Sub-Tasks
1. - [ ] Name: Define Drive Schemas
   - Description: Design `drive_folders` and `drive_files` tables incl. indexes (owner, updated_at).
   - Original Estimation: 6h
2. - [ ] Name: OAuth & Service Setup
   - Description: Configure GDrive API client, token storage/refresh, minimal permissions.
   - Original Estimation: 6h
3. - [ ] Name: Incremental Sync Job
   - Description: Implement change-token based sync for folders/files; upsert logic.
   - Original Estimation: 10h
4. - [ ] Name: Revisions & Attachments
   - Description: Capture file revisions and attachment metadata for downstream RAG.
   - Original Estimation: 8h
5. - [ ] Name: Logging & Metrics
   - Description: Write sync entries; expose basic metrics (counts, latency, failures).
   - Original Estimation: 4h

### Story S3: Gmail Connector - Mail/Thread Sync
- [ ] Story Status
- Story Points: 8
- Acceptance Criteria (Documentation):
  - Sync emails, threads, labels, and metadata (`to`, `cc`, `bcc`, `subject`, `message_id`).
  - Attachments captured and stored; incremental sync by historyId.
  - Errors retried; rate limits respected; sync logged.
  - Data persists in `mail_threads`, `mail_messages`, `mail_attachments` tables.
- Related Docs: features/gmail.md

#### Sub-Tasks
1. - [ ] Name: Define Mail Schemas
   - Description: Create `mail_threads`, `mail_messages`, `mail_attachments` with indexes.
   - Original Estimation: 6h
2. - [ ] Name: Gmail API Client Setup
   - Description: OAuth scopes, token storage, API plumbing, rate-limit handling.
   - Original Estimation: 6h
3. - [ ] Name: Incremental Mail Sync
   - Description: Use historyId to pull new messages, upsert threads/messages.
   - Original Estimation: 10h
4. - [ ] Name: Attachment Handling
   - Description: Download/store metadata, link attachments to messages.
   - Original Estimation: 6h
5. - [ ] Name: Sync Logging & Metrics
   - Description: Structured logs and counters for synced messages/threads and failures.
   - Original Estimation: 4h

### Story S4: GChat Connector - Space/Member/Message Sync
- [ ] Story Status
- Story Points: 8
- Acceptance Criteria (Documentation):
  - Sync spaces, members/roles, messages, threads, and attachments; track message owners.
  - Incremental sync and idempotent upserts; error retries with backoff.
  - Data persists in `chat_spaces`, `chat_members`, `chat_messages` tables.
- Related Docs: features/gchat.md

#### Sub-Tasks
1. - [ ] Name: Define Chat Schemas
   - Description: Create `chat_spaces`, `chat_members`, `chat_messages` with indexes.
   - Original Estimation: 6h
2. - [ ] Name: GChat API Setup
   - Description: OAuth scopes, service client config, pagination handling.
   - Original Estimation: 6h
3. - [ ] Name: Messages & Threads Sync
   - Description: Pull messages/threads; upsert with owners and attachments.
   - Original Estimation: 10h
4. - [ ] Name: Members & Roles Sync
   - Description: Sync space membership and roles with deltas.
   - Original Estimation: 6h
5. - [ ] Name: Logging & Metrics
   - Description: Structured logs and metrics for spaces/members/messages sync.
   - Original Estimation: 4h

### Story S5: Database Connector - NL2SQL Querying
- [ ] Story Status
- Story Points: 8
- Acceptance Criteria (Documentation):
  - NL queries translated into safe SQL with schema-aware constraints.
  - Parameterization and read-only enforcement; results formatted with pagination.
  - Errors handled gracefully; audit logs captured.
- Related Docs: features/database.md

#### Sub-Tasks
1. - [ ] Name: Schema Introspection
   - Description: Catalog table/column metadata to guide NL2SQL.
   - Original Estimation: 6h
2. - [ ] Name: NL2SQL Prompting Layer
   - Description: Implement prompts and heuristics to produce safe SQL.
   - Original Estimation: 8h
3. - [ ] Name: SQL Execution Engine
   - Description: Parameterized queries, pagination, type mapping.
   - Original Estimation: 8h
4. - [ ] Name: Result Formatter
   - Description: Tabular/JSON outputs with limits and summaries.
   - Original Estimation: 6h
5. - [ ] Name: Audit Logging
   - Description: Record query text, SQL, execution time, and requester.
   - Original Estimation: 4h

### Story S6: Web Connector - Enterprise Web Search
- [ ] Story Status
- Story Points: 5
- Acceptance Criteria (Documentation):
  - Perform web searches with domain allowlist; store snippets and sources.
  - Relevance ranking and basic credibility checks; cache results.
  - Clearly mark external content with citations.
- Related Docs: features/web-connector.md

#### Sub-Tasks
1. - [ ] Name: Search API Integration
   - Description: Integrate with a web search provider; handle quotas.
   - Original Estimation: 4h
2. - [ ] Name: Relevance Ranking
   - Description: Score results and filter low-credibility domains.
   - Original Estimation: 6h
3. - [ ] Name: Caching Layer
   - Description: Cache queries and top results with expiry.
   - Original Estimation: 4h
4. - [ ] Name: Citations & Storage
   - Description: Persist snippets and source URLs for RAG usage.
   - Original Estimation: 4h
5. - [ ] Name: Logging & Metrics
   - Description: Track queries, cache hit rate, and result quality signals.
   - Original Estimation: 3h

### Story S7: Conversation & Feedback Management
- [ ] Story Status
- Story Points: 5
- Acceptance Criteria (Documentation):
  - Persist conversation sessions and messages; maintain context across sessions.
  - Feedback ratings (1–5) stored with message references.
  - APIs for listing conversations, posting messages, and rating.
- Related Docs: features/conversation-feedback.md

#### Sub-Tasks
1. - [ ] Name: Conversation Schemas
   - Description: Tables for sessions, messages, feedback with indexes.
   - Original Estimation: 4h
2. - [ ] Name: Context Manager
   - Description: Rehydrate conversation context for agent calls.
   - Original Estimation: 5h
3. - [ ] Name: Feedback API
   - Description: Endpoints to submit and fetch ratings.
   - Original Estimation: 4h
4. - [ ] Name: History APIs
   - Description: List sessions/messages with filters and limits.
   - Original Estimation: 4h
5. - [ ] Name: Privacy Controls
   - Description: Redaction options and retention settings for conversations.
   - Original Estimation: 3h

### Story S8: Schedulers - Sync Orchestration & Logging
- [ ] Story Status
- Story Points: 5
- Acceptance Criteria (Documentation):
  - Central scheduler triggers connector syncs with retry/backoff.
  - All syncs write entries to `sync_logs` with timing and counts.
  - Configurable schedules per connector.
- Related Docs: features/schedulers.md

#### Sub-Tasks
1. - [ ] Name: Scheduler Module
   - Description: Implement job scheduler interface & runner.
   - Original Estimation: 4h
2. - [ ] Name: Connector Tasks
   - Description: Define per-connector jobs and parameters.
   - Original Estimation: 4h
3. - [ ] Name: Retry & Backoff
   - Description: Add exponential backoff with jitter and max attempts.
   - Original Estimation: 4h
4. - [ ] Name: Sync Logs Table
   - Description: Create `sync_logs` schema and write entries.
   - Original Estimation: 3h
5. - [ ] Name: Config & Cron
   - Description: Externalize schedules and enable per-env setup.
   - Original Estimation: 3h

### Story S9: User Management - Auth & RBAC
- [ ] Story Status
- Story Points: 5
- Acceptance Criteria (Documentation):
  - OAuth login, session management, and role assignment.
  - RBAC enforcement across connectors, RAG retrieval, and APIs.
  - Admin UI/API for user and role management.
- Related Docs: features/user-management.md

#### Sub-Tasks
1. - [ ] Name: OAuth & Sessions
   - Description: Implement Google OAuth, session tokens, refresh.
   - Original Estimation: 5h
2. - [ ] Name: RBAC Model
   - Description: Define roles/permissions and persistence schema.
   - Original Estimation: 4h
3. - [ ] Name: Enforcement Middleware
   - Description: Gate APIs and RAG calls by role/ACL.
   - Original Estimation: 5h
4. - [ ] Name: Admin APIs
   - Description: Manage users, roles, invites.
   - Original Estimation: 4h
5. - [ ] Name: Audit Hooks
   - Description: Log auth events and permission checks.
   - Original Estimation: 3h

### Story S10: Governance - Data Access & Compliance
- [ ] Story Status
- Story Points: 5
- Acceptance Criteria (Documentation):
  - Documented privilege model and data governance policies.
  - Enforcement at connector, RAG retrieval, and API layers.
  - PII redaction and retention controls.
- Related Docs: features/governance.md

#### Sub-Tasks
1. - [ ] Name: Governance Policy Doc
   - Description: Draft policies for data access, retention, and audit.
   - Original Estimation: 4h
2. - [ ] Name: ACL Schema
   - Description: Define ACL structures stored with data chunks.
   - Original Estimation: 4h
3. - [ ] Name: Enforcement in RAG
   - Description: Apply ACL filters during retrieval.
   - Original Estimation: 4h
4. - [ ] Name: Retention Jobs
   - Description: Scheduled cleanup per policy.
   - Original Estimation: 3h
5. - [ ] Name: Compliance Logs
   - Description: Track sensitive access and policy decisions.
   - Original Estimation: 3h

### Story S11: Enterprise Brain Agent - Query Resolution
- [ ] Story Status
- Story Points: 8
- Acceptance Criteria (Documentation):
  - Agent parses queries, invokes RAG pipeline, and returns structured answers.
  - Supports streaming responses and clear citations; graceful fallback when low context.
  - Answers adhere to governance and RBAC.
- Related Docs: features/enterprise-brain-agent.md

#### Sub-Tasks
1. - [ ] Name: Agent Prompting
   - Description: Design system/user prompts and tools schema.
   - Original Estimation: 6h
2. - [ ] Name: RAG Tool Adapter
   - Description: Integrate agent with RAG service SDK/API.
   - Original Estimation: 6h
3. - [ ] Name: Response Formatter
   - Description: Structure answers with citations and confidence.
   - Original Estimation: 5h
4. - [ ] Name: Fallback Handling
   - Description: Default replies when context is insufficient.
   - Original Estimation: 4h
5. - [ ] Name: Streaming Support
   - Description: Stream tokens to client; handle cancellations.
   - Original Estimation: 5h

### Story S12: Implement Schema-Aware Agent Query Orchestration
- [ ] Story Status
- Story Points: 8
- Acceptance Criteria (Documentation):
  - Agent can plan and execute schema-aware SQL, vector, and RAG queries.
  - Workflows are deterministic, idempotent, and traceable.
  - Logs and metrics are emitted for each step.
- Related Docs: features/orchestration.md

#### Sub-Tasks
1. - [ ] Name: Schema Loading and Parsing
   - Description: Load and parse the application schema (`app-schema.sql`) at agent startup. Validate the schema structure and convert it into a structured JSON format for agent consumption.
   - Original Estimation: 6h
2. - [ ] Name: Semantic Schema Registry
   - Description: Build a semantic schema registry enriched with business semantics. This registry will act as the single source of truth for table purposes, searchable fields, vector fields, and timestamp fields.
   - Original Estimation: 8h
3. - [ ] Name: Query Strategy Classification
   - Description: Implement a Planning Agent that produces a detailed, structured execution plan using the semantic schema registry. The plan includes `strategy` and a sequence of tasks with `id`, `goal`, `task` payload (e.g., SQL filter, vector search, RAG generate), `constraints` (allowed tables/columns, LIMIT, time bounds, read-only), `confidence`, and `next_criteria` for dependency-based decision making. The plan is consumed by a React Agent which executes via tools. No query generation/execution in this step.
   - Original Estimation: 6h
4. - [ ] Name: Schema-Aware SQL Query Builder
   - Description: Implement a SQL query builder that generates schema-aware SQL queries strictly constrained by the schema registry. Ensure queries are read-only and enforce limits.
   - Original Estimation: 8h
5. - [ ] Name: Vector Query Builder
   - Description: Implement a vector query builder that generates similarity queries using vector fields defined in the schema registry. Support hybrid execution paths where vector queries are scoped to SQL-filtered results.
   - Original Estimation: 8h
6. - [ ] Name: Hybrid Query Execution Flow
   - Description: Orchestrate the execution of hybrid queries in the following order: SQL filtering → vector ranking → RAG interpretation. Maintain intermediate results for traceability and ensure each step consumes outputs from the previous step.
   - Original Estimation: 10h
7. - [ ] Name: Query Validation Layer
   - Description: Implement a validation layer to ensure all generated queries are valid before execution. Validate against the schema registry and enforce constraints like `LIMIT` and read-only operations.
   - Original Estimation: 6h
8. - [ ] Name: RAG Builder over Query Results
   - Description: Build RAG prompts using only the outputs of SQL and vector queries. Prevent blind vector retrieval or full-store retrieval. Pass document IDs and content explicitly to the RAG layer.
   - Original Estimation: 8h
9. - [ ] Name: Answer Composition with Evidence
   - Description: Generate the final answer using RAG output and include supporting evidence references (e.g., email IDs, message IDs, file IDs). Ensure no facts are introduced that are not present in the evidence.
   - Original Estimation: 6h
10. - [ ] Name: End-to-End Orchestration Wiring
    - Description: Wire all components into a single deterministic execution pipeline. Ensure reproducibility, debuggability, and structured logging at each step.
    - Original Estimation: 10h

### Story S13: Logging & Audit Trails
- [ ] Story Status
- Story Points: 3
- Acceptance Criteria (Documentation):
  - Structured logs across connectors, RAG, and agent.
  - Audit entries for user actions and data access.
  - Log routing to files/DB; retention configured.
- Related Docs: features/logging.md

#### Sub-Tasks
1. - [ ] Name: Log Format & Sinks
   - Description: Define JSON log format and sinks (stdout, file, DB).
   - Original Estimation: 3h
2. - [ ] Name: Audit Table Schema
   - Description: Create audit tables and write hooks.
   - Original Estimation: 3h
3. - [ ] Name: Log Routing
   - Description: Configure loggers and handlers per module.
   - Original Estimation: 2h
4. - [ ] Name: Dashboards
   - Description: Basic dashboards for error rates and throughput.
   - Original Estimation: 3h
5. - [ ] Name: Retention Policies
   - Description: Configure log rotation and retention.
   - Original Estimation: 2h

### Story S14: LLM Monitoring & Telemetry
- [ ] Story Status
- Story Points: 3
- Acceptance Criteria (Documentation):
  - Monitor LLM token usage, latency, and cost; per-user metrics.
  - Alerts for anomalies; sampling for quality checks.
  - Dashboards summarizing usage and performance.
- Related Docs: features/llm-monitoring.md

#### Sub-Tasks
1. - [ ] Name: Token & Latency Metrics
   - Description: Capture token counts and timings per request.
   - Original Estimation: 3h
2. - [ ] Name: Cost Estimator
   - Description: Estimate cost per request and aggregate.
   - Original Estimation: 2h
3. - [ ] Name: Usage Dashboards
   - Description: Build dashboards for per-user/org usage.
   - Original Estimation: 3h
4. - [ ] Name: Alerts & Thresholds
   - Description: Configure alerts for spikes/latency/cost.
   - Original Estimation: 2h
5. - [ ] Name: Quality Sampling
   - Description: Sample responses for periodic quality review.
   - Original Estimation: 3h