# Real-Time Decision Intelligence – Data Flow & Processing ## 1. Execution Flow ### Step 1 — Trigger Refresh The Refresh Mechanism fires (scheduled or event-driven) and calls the Data Source Agent to initiate the sync cycle. ### Step 2 — Data Synchronization The Data Source Agent connects to Gmail and fetches the latest emails for each user. Each user's data is synced independently and stored in the Gmail tables. ### Step 3 — Data Processing The Data Source Agent processes the synced data and prepares it for the RT Agent. This step differs based on the chosen approach — see Section 2 (user-level) or Section 3 (project-level). ### Step 4 — RT Agent Aggregation The RT Agent consumes the prepared data, aggregates it across users and sources, and produces a unified project-level snapshot. ### Step 5 — Snapshot Update The RT Agent pushes the finalized snapshot to the Org Snapshot / Knowledge Graph. The organizational view is updated with the latest project intelligence. --- ## 2. Approach 1: User-Level Processing → RT Agent Aggregation In this approach, the Data Source Agent processes data at the **individual user level** and passes per-user summaries to the RT Agent. The RT Agent is responsible for aggregating these into a project-level view. ### 2.1 Data Source Agent — Per-User Processing For each user, the agent: - Extracts all new emails from the Gmail tables - Generates a summary for each email - Enriches each email with: - **Participants** — To, CC, BCC (who was involved in the email) - **Project tag** — which project this email relates to (e.g., `#EnterpriseBrain`) - **Type** — a single flag classifying the signal: `information`, `alert`, `insight`, or `escalation` - **Categories** — classification of the email type (e.g., escalation, update, bug report, release) - **Hashtags** — additional keyword signals (e.g., `#Release`, `#Urgent`, `#Q2`) **Output per user:** A list of enriched email summaries, each carrying participants, project tag, type flag, categories, and hashtags. --- ### 2.2 RT Agent — Project-Level Aggregation The RT Agent receives all per-user summaries and groups them by project. **For each project (e.g., Enterprise Brain):** - Consolidates all emails tagged to the project across all users - Builds a participant map — who is involved at what access level - Applies access control — determines which email or detail is visible to which role - Retains categories and hashtags from the per-user summaries --- ### 2.3 Multi-Source Aggregation When multiple data sources are active (e.g., Gmail and Jira), the RT Agent consumes enriched data from both and merges them at the project level. **Final project-level snapshot includes:** - **Gmail:** email summaries, participants, categories, hashtags - **Jira:** ticket updates, assignees, status changes, labels - **Unified view:** cross-source activity per project, access-controlled --- ## 3. Approach 2: Direct Project-Level Processing in Data Source Agent In this approach, the Data Source Agent skips the per-user summary step and generates **project-level insights** directly. The RT Agent's role is reduced to validation, merging across sources, and finalization. ### 3.1 Data Source Agent — Direct Project-Level Output Instead of producing per-user summaries, the agent aggregates across all users and groups everything by project upfront. **For each project (e.g., Enterprise Brain), the agent produces:** - All emails received across users that relate to this project - Consolidated participant list across users - Access mapping — which user or role can view which email - A type flag per email — `information`, `alert`, `insight`, or `escalation` - Categories and hashtags applied across the email set The agent effectively performs the aggregation work that the RT Agent would otherwise own in Approach 1. --- ### 3.2 RT Agent — Reduced Role With project-level data already prepared by the Data Source Agent, the RT Agent: - Validates and deduplicates the incoming data - Merges insights across multiple sources (e.g., Gmail output + Jira output for the same project) - Finalizes the snapshot and pushes it to the Org Snapshot --- ## 4. RT Agent Output — Per-Project Snapshot to Org Knowledge Graph Regardless of which approach is used, the RT Agent's final responsibility is the same: send a structured per-project snapshot to the Org Knowledge Graph. Each snapshot consolidates all data sources for that project into a single object. ```json { "project": "Enterprise Brain", "last_updated": "2026-03-19T10:32:00Z", "sources": { "gmail": { "total_emails": 12, "emails": [ { "id": "email_001", "summary": "Release date pushed to March 25", "type": "information", "participants": ["Alice", "Bob", "PM-lead"], "category": "release-update", "hashtags": ["#Release", "#Q1"], "access": ["Alice", "Bob", "PM-lead"] }, { "id": "email_002", "summary": "Bug reported in login flow", "type": "alert", "participants": ["Alice", "Dev-lead"], "category": "bug-report", "hashtags": ["#Bug", "#Urgent"], "access": ["Alice", "Dev-lead"] } ] }, "jira": { "total_tickets": 5, "tickets": [ { "id": "DEB-101", "summary": "Login flow crash on Safari", "assignee": "Dev-lead", "status": "In Progress", "labels": ["#Bug", "#Sprint3"] } ] } }, "participants": ["Alice", "Bob", "Carol", "PM-lead", "Dev-lead"], "categories": { "release-update": 5, "bug-report": 4, "general": 3 }, "hashtags": ["#Release", "#Bug", "#Urgent", "#Q1", "#Sprint3"] } ``` This is what gets stored in the Org Snapshot for each project. When a user queries Enterprise Brain from the decision center, this snapshot is what backs the response. --- ## 5. Approach Comparison ### Approach 1 — User-Level → RT Agent Aggregation ```mermaid flowchart TD A([Refresh Trigger]) --> B[Data Source Agent] B --> G1[(Gmail Tables\nUser: Alice)] B --> G2[(Gmail Tables\nUser: Bob)] B --> G3[(Gmail Tables\nUser: Carol)] G1 --> P1[Per-User Processing\nAlice: summarise + enrich emails] G2 --> P2[Per-User Processing\nBob: summarise + enrich emails] G3 --> P3[Per-User Processing\nCarol: summarise + enrich emails] P1 --> RT[RT Agent] P2 --> RT P3 --> RT RT --> AGG[Aggregate by Project\n+ Build participant map\n+ Apply access control] AGG --> SNAP[(Org Snapshot /\nKnowledge Graph)] ``` ### Approach 2 — Direct Project-Level in Data Source Agent ```mermaid flowchart TD A([Refresh Trigger]) --> B[Data Source Agent] B --> G1[(Gmail Tables\nUser: Alice)] B --> G2[(Gmail Tables\nUser: Bob)] B --> G3[(Gmail Tables\nUser: Carol)] G1 --> AGG[Aggregate Directly\nby Project across all users] G2 --> AGG G3 --> AGG AGG --> RT[RT Agent] RT --> VAL[Validate + Deduplicate\n+ Merge multi-source] VAL --> SNAP[(Org Snapshot /\nKnowledge Graph)] ``` **Recommendation:** Approach 1 is better suited for a multi-source architecture (Gmail + Jira + future connectors) because the RT Agent is the natural owner of cross-source project aggregation. Approach 2 may be acceptable for early-stage builds with a single data source. --- ## 6. Example Walkthrough **Scenario:** 3 users — Alice, Bob, and Carol — are part of the Enterprise Brain project. A refresh cycle is triggered. **Step 1–2:** The Data Source Agent syncs Gmail for all three users. Their inboxes are fetched and stored in the Gmail tables: | User | Emails fetched | |-------|----------------| | Alice | 4 new emails | | Bob | 2 new emails | | Carol | 6 new emails | **Step 3 (Approach 1 — user-level):** For each user, the agent generates a summary per email. Each user becomes an object containing their email objects: ```json { "user": "Alice", "emails": [ { "id": "email_001", "summary": "Release date pushed to March 25", "type": "information", "participants": ["Alice", "Bob", "PM-lead"], "project": "#EnterpriseBrain", "category": "release-update", "hashtags": ["#Release", "#Q1"] }, { "id": "email_002", "summary": "Bug reported in login flow", "type": "alert", "participants": ["Alice", "Dev-lead"], "project": "#EnterpriseBrain", "category": "bug-report", "hashtags": ["#Bug", "#Urgent"] } ] } ``` All 3 user objects (Alice with 4 emails, Bob with 2, Carol with 6) are passed to the RT Agent. **Step 4:** The RT Agent groups emails by project. For `#EnterpriseBrain`, it consolidates all 12 summaries, builds a participant map across Alice, Bob, and Carol, applies access rules, and produces a single project snapshot. **Step 5:** The snapshot is written to the Org Knowledge Graph: ``` Project: Enterprise Brain Sources: Gmail Emails: 12 (release updates, bug reports) Participants: Alice, Bob, Carol, PM-lead, ... Categories: release-update (5), bug-report (3), general (4) Last updated: 2026-03-19T10:32:00Z ``` --- ## 7. Key Concepts | Term | Definition | |---|---| | **Insight** | A derived understanding from raw data — e.g., "Project X has had 3 escalations this week" | | **Snapshot** | A point-in-time structured view of all project-level data, stored in the Org Knowledge Graph | | **Category** | A logical classification of a data item — e.g., bug report, escalation, release update | | **Hashtag / Tag** | A keyword signal attached to a data item to aid grouping and search — e.g., `#EnterpriseBrain`, `#Q2` | | **Participant map** | A record of who is involved in a project communication, derived from To/CC/BCC fields | | **Access control** | Rules governing which user or role can view a given email, ticket, or insight within a project | --- ## 8. Open Items Things that need to be nailed down before implementation: 1. **Email extraction depth** — Decide exactly how we break down an email to extract maximum information. This includes what fields we parse (subject, body, thread context, attachments), how we handle forwarded or quoted content, and how granular the summary should be. 2. **Duplicate email handling** — When the same email (e.g., a meeting notes blast) is received by multiple users without explicit CC relationships, we need a smarter deduplication strategy. Pure content matching may not be enough; we need to decide the right identity key for an email across inboxes. 3. **Standardising categories, hashtags, and tags** — The current categories and hashtags are loosely defined. We need a fixed taxonomy: what categories are valid, who defines new ones, whether tags are AI-generated or user-defined, and how conflicts or overlaps are resolved. 4. **Project identification per email** — We need a reliable way to determine which project an email belongs to. An email does not always mention a project by name — it could be inferred from sender domain, subject keywords, thread history, or participant list. The method must be consistent and handle ambiguous cases (e.g., an email that touches multiple projects). 5. **Email body storage vs. security** — For security and privacy reasons, we should not store the raw email body in the Gmail tables. However, the Data Source Agent and RT Agent need the body content to generate summaries, derive type, categories, hashtags, and other insights. Without the body being stored, how do we provide this capability securely? Some options to evaluate: - **Process in memory only** — fetch the body live from Gmail API, process it immediately in memory to extract all derived fields, then discard it without ever writing to the database. Derived fields (summary, type, category etc.) are what get stored, never the body itself. - **Encrypted storage** — store the body encrypted at rest with per-user keys, decrypt only during processing, ensuring the raw content is never exposed in plaintext. - **Tokenised reference** — store only the Gmail message link and re-fetch body on demand when processing is needed, keeping the body outside our system entirely.