--- meeting: Enterprise Brain Scrum date: 2026-04-02 participants: - Yeshwanth Reddy Yerraguntla - Rajashekar G - Vara Kumar Jagarapu duration: ~17:00 transcript: 20260402.txt --- # Enterprise Brain Scrum — April 2, 2026 A brief scrum addressing stakeholder feedback concerns, branch strategy for the Tata Steel integration, evaluation planning with Promptfoo, and the methodology for ensuring agent answer reliability through consistency testing. The meeting emphasized the importance of getting the evaluation question dataset right before worrying about evaluation mechanisms. ## Act I - Stakeholder Feedback and System Improvements (~00:00:00 - ~00:05:29) Rajashekar reported that Yeshwanth's comments on Abhilash's document have been noted and the team will ensure proper indentation. More critically, he flagged that stakeholders—specifically Navin—aren't happy with the current outputs from the data source agent. > _"I think they are not happy with the outputs right now. Whatever we are getting on data official approved."_ > — Rajashekar G _(~00:03:17)_ Everything is being logged to the database, so Rajashekar plans to review the logs and correct the system prompts. Currently, the agent receives no hand-fed business logic—it's operating purely based on schema descriptions (table names and their descriptions). The improvement path: inject proper business context and logic into the system. Yeshwanth asked for qualitative feedback beyond "they didn't like it." Rajashekar committed to getting specifics from Navin in office. On the design front, Amulya is working with the design branch. Rajashekar will coordinate with Vara Kumar to finalize the design system changes. ## Act II - Branch Strategy and Quality Concerns (~00:05:29 - ~00:07:06) Rajashekar explained the branch architecture: he created a new branch called `chanakya/BA` containing only the RT agent and Tata Steel data source agent (two ports only) plus all UI and backend implementation. A PR is open to `chanakya/alpha`, but they're deliberately not merging immediately—everything must be verified first. > _"I have a PR to chica/ alpha. We not right away merging it but we need to verify everything."_ > — Rajashekar G _(~00:05:29)_ Vara Kumar mentioned "boiler plate" and quality concerns—stakeholders want quality improvements. Prabhakar apparently had specific points that will lead to better results. The team plans to sync in the evening for detailed discussions. ## Act III - Evaluation Strategy and Consistency Testing (~00:07:06 - ~00:14:30) The evaluation discussion began with Rajashekar mentioning Prabhakar's work yesterday connecting the system with Promptfoo for evaluation. Two evaluation tracks emerged: 1. **Code evaluation**: Unit tests for both frontend and backend 2. **Agent evaluation**: A mechanism to evaluate agent answer quality Vara Kumar re-introduced Navin's reliability testing concept: use multiple agents (actually multiple LLMs) to answer the same question. If three LLMs give the same answer, confidence in that answer increases. If there's contradiction, something is wrong with one of the agents. The medical example illustrated the stakes: customers care less about incomplete answers than wrong answers. For critical domains, wrong information is dangerous. > _"they're not worried about if they didn't get some I mean complete answer but they're worried about a wrong answer right in that case what he mentioned for if that something agent is related to some medical thing so that should be very accurate"_ > — Vara Kumar Jagarapu _(~00:08:26)_ Yeshwanth sought clarification: they're talking about LLMs, not agents. Vara Kumar confirmed—ask the same question to three LLMs. If all three agree, consider the first one's answer reliable. Yeshwanth acknowledged consistency evaluation as a valid LLM evaluation paradigm but redirected focus to the critical unknown: what questions are being asked? > _"the major unknown here is what are the questions we are asking. So Rajika irrespective of what testing method we take up right I would say don't even worry about which method focus only on which uh questions are we going to ask and how it should answer that is critical once you have the data set uh asking questions evaluating is literally two three hours job focus only on"_ > — Yeshwanth Reddy Yerraguntla _(~00:11:40)_ Rajashekar expressed the same worry: questions are the bottleneck. Previously Solv.it provided questions that generated correct answers in current deployment, but the business intent from Navin and Prabhakar may differ. Yeshwanth reframed this as an accountability issue, not a quality issue: > _"You you evaluated it and it is working. It's just that what we evaluated against is not what they were expecting. So get a sign off from Prabhakar with the same questions. Simple. We'll escalate this saying this is what we tested on. This is what was the report. If there is a gap then the gap is uh in our understanding not in the execution."_ > — Yeshwanth Reddy Yerraguntla _(~00:11:40)_ Rajashekar also flagged data quality issues: many columns contain null values, affecting answer generation. These gaps need investigation and discussion with stakeholders today to determine where improvements are needed. Yeshwanth asked about specific gaps. Rajashekar began screen sharing to show examples, pointing to instances where data values were simply marked "same as above" instead of containing actual values. These nulls weren't caught in the first iteration. The team wrote queries to update missing values in the database. ## Todos - [ ] Review database logs and inject business logic into system prompts to improve output quality | author: Rajashekar G | deadline: 2026-04-03 - [ ] Get qualitative feedback from Navin on specific output quality issues | author: Rajashekar G | deadline: 2026-04-02 - [ ] Coordinate with Vara Kumar to finalize design system changes and merge design branch work | author: Rajashekar G, Amulya | deadline: 2026-04-03 - [ ] Verify all changes in chanakya/BA branch before merging PR to chanakya/alpha | author: Rajashekar G, Team | deadline: 2026-04-03 - [ ] Get stakeholder sign-off on evaluation question dataset from Prabhakar | author: Rajashekar G | deadline: 2026-04-02 - [ ] Document and escalate data quality gaps (null values, "same as above" placeholders) with stakeholders | author: Rajashekar G | deadline: 2026-04-02 - [ ] Continue Promptfoo evaluation setup and connect both code and agent evaluation tracks | author: Prabhakar, Team | deadline: 2026-04-04