A Singapore Government Agency Website
How to identify
The Instant Messaging Analyzer Tool enables SEAB Exam Operations Department Officers to efficiently analyze and categorize daily exam queries, reducing reliance on individual judgment and enhancing data-driven decision-making for 80,000 candidates across 325 examination centres.
Team members and respective divisions: Keith Lim (MOE ITD), Tze Eng (MOE ITD), Kristoffer Liau (MOE ITD), Jin Yuen Quek (Trusted Infrastructure & Services/GPE), Sreenivasa Durga Prasad (MOE ITD)
Details on our problem statements and problem formulation process:
Initial Problem Statement:
Exams Operations Department (EOD) Officers from SEAB (Singapore Examinations and Assessment Board) need to evaluate the potential impact of an automated ChatBot in enhancing efficiency and user experience for queries from public and internal users via instant messaging channels (such as WhatsApp and Telegram), quantifying the extent of possible improvements when administering the conduct the GCE and PSLE National exams in the SEAB Ops Room.
But SEAB Officers lack both the capacity and the analytical tools to efficiently analyze over 20,000 instant messages from 700 Exam Personnel (School Teachers), hindering their ability to evaluate the potential impact of an automated ChatBot in enhancing efficiency and user experience for these queries. This limitation prevents them from quantifying the extent of possible improvements and identifying frequently asked queries suitable for automation across instant messaging channels.
This results in a critical need to leverage data analytical technology to identify frequently asked queries suitable for ChatBot automation and quantify its potential impact. Such technology would enable any agency to efficiently analyse large volumes of instant messages, determining the viability and benefits of a ChatBot solution for enhancing efficiency and user experience. This matters because it allows an agency to make an informed, data-driven decision about implementing a potentially time-saving and efficiency-boosting ChatBot for all stakeholders (in SEAB's case: 50 SEAB Officers and 700 Exam Personnel) across various instant messaging channels.
Problem Formulation Process:
1.Primary Research and Analysis:
To gain a comprehensive understanding of the operational challenges, we conducted
extensive user research with EOD officers who manage daily examination-related queries.
Through in-depth interviews and process observation, we identified several critical operational
pain points that significantly impact the department's efficiency:
• Manual Query Processing and Identification:
Officers spend considerable time scrolling through numerous WhatsApp conversations to identify frequent or important queries. This manual process is not only time-consuming but also relies heavily on individual judgment to determine which queries require immediate attention or indicate emerging trends.
• Inconsistent Query Categorisation:
The absence of a standardised categorisation framework leads to varying interpretations of
similar queries by different officers. This inconsistency makes it difficult to
aggregate data meaningfully and track patterns across the examination centres.
• Resource-Intensive Report Generation:
The compilation of daily and periodic reports requires officers to manually collate and
summarise query data, a process that diverts significant time away from addressing
urgent operational matters.
• Peak Period Resource Strain:
During examination periods, the volume of queries increases substantially, putting immense
pressure on the limited number of officers. This strain often leads to delayed responses.
• Limited Trend Analysis Capabilities:
Without systematic data collection and analysis tools, officers struggle to identify long-term
patterns or emerging issues that could inform proactive operational improvements.
2.Problem Refinement
• The problem focus underwent a significant shift during this research phase. Initially, the scope
centered on assessing the impact of implementing a ChatBot and evaluating the benefits of
automation. However, user research revealed deeper operational challenges. This led to
refining the problem statement to address the core challenge of efficient
query analysis, emphasizing the manual and subjective nature of current processes, the strain
on limited manpower, and the inability to track trends objectively.
• By resolving these fundamental challenges through automated analysis and structured
categorisation, the resulting insights would also provide the essential foundation needed
for futureChatBot implementation – including understanding common query patterns,
standardised responses, and peak timing of different query types.
3.Root Cause Analysis
Through detailed examination of the operational challenges and user feedback, we conducted a comprehensive analysis to identify the fundamental factors contributing to the current inefficiencies in query management. Our root cause analysis revealed several interconnected issues that needed to be addressed:
• Absence of automated data extraction tools:
The lack of automated tools forces officers to manually extract and compile information from
WhatsApp conversations, resulting in significant time investment and potential for human error.
• Lack of structured query categorisation system:
Without a standardised system for categorising queries, officers rely on individual
interpretation, leading to inconsistent classification and difficulties in tracking trends accurately.
• Limited ability to identify and track recurring issues:
The current manual process makes it challenging to spot patterns and recurring problems
effectively, preventing proactive problem-solving and operational improvements.
• Inefficient use of officer time on manual processes:
Valuable officer time is spent on repetitive tasks like scrolling through messages and compiling
reports, rather than on strategic analysis and problem-solving.
• Difficulty in generating objective insights for operational improvements:
The combination of manual processing and subjective categorisation makes it challenging to
generate reliable data-driven insights that could inform operational enhancements.
Final Problem Statement:
SEAB's Exam Operations Department struggles to efficiently analyze thousands of daily queries from 700 exam personnel across 325 centres. Without structured analysis, critical patterns are missed and inconsistent categorization prevents timely targeted improvements, impacting smooth operations for exam personnel while managing examinations for 80,000 candidates.
Final How Might We Statement:
How might we automate the extraction and analysis of queries from SEAB's WhatsApp Personal account to improve consistency, efficiency, and data-driven decision-making for daily operations and post-exam trend reporting?
Details of our Solution:
The Instant Messaging Analyser Tool (IMAT) is an innovative system designed to transform how SEAB processes and derives insights from WhatsApp conversations. This comprehensive solution integrates advanced data processing, artificial intelligence, and interactive visualization capabilities through three primary subsystems:
Data Extraction and Cleansing Subsystem - Our first component focuses on secure and efficient data processing, incorporating:
• Extracts raw chat messages from WhatsApp conversations • Processes the data through an anonymisation batch job to remove sensitive information • Cleanses the data by removing redundant responses and/or irrelevant content • Prepares standardised data for analysis
Analyser Engine - The core processing component leverages cutting-edge technology to deliver comprehensive analysis:
• Processes the cleansed chat data through categorisation algorithms • Generates comprehensive summaries of conversation based on query or topics • Identifies patterns and trends in the chat conversations • Produces actionable insights based on message content
Interactive Dashboard - A user-centric interface designed for accessibility and functionality:
• Presents analysis results through a web-based, mobile responsive interface • Displays key metrics including: o Top recurring queries o Frequently asked questions o Temporal analysis breakdowns (daily, weekly, monthly, and yearly views) o Chat Volume and Trend visualisations • Provides actionable recommendations for service improvements based on chat analysis • Enables users to explore data through interactive visualisations
Impact and outcomes analysis of our solution:
• Automated categorization and insights enable near real-time improvements by leveraging a
Large Language Model (LLM) to classify queries and generate objective insights efficiently.
• End-to-end automation of WhatsApp query processing, from secure extraction via a locally
installed tool to data transformation, ensures structured and accurate analysis.
• Data-driven decision-making reduces recurring queries, enhances exam efficiency, and
minimizes reliance on manual judgment.
• Reduction in manual workload, saving officer hours by automating query processing, analysis,
and reporting.
• Lower operational costs by minimizing recurring issues and streamlining reporting through a
web-based report generator, enabling quick access to trends, top queries, and statistics for
targeted improvements.
Future steps for our project:
• Pilot with SEAB EOD Officers during the next exam cycle to refine query categorization and insights. • Scale implementation to support broader exam operations across SEAB if the pilot proves successful. • Gather feedback and refine features based on real-world usage.