DataSharingAssist: Making Data Sharing Stress-Free to Foster Openness and Progress

The Data Sharing Nightmare: Problem Statement

Public officers are required to assess data quality, privacy risks, and data classification when evaluating their agency's datasets, typically requested through the Standardised Data Submission Form (SDSF)—a process that is often challenging and inefficient, as confirmed by our user research.

Challenges:

• Complex IM8 guidelines: Difficult to interpret and apply consistently.

• Uncertainty in dataset quality and privacy risks: Lack of clear evaluation criteria.

• Limited awareness of Whole-of-Government (WOG) tools: Existing solutions for managing privacy risks are underutilized.

• No standardized approach: Challenges in down-classifying or cleaning datasets for different use cases.

Impact:

• Delays & inefficiencies: Repeated back-and-forth emails with requesters due to unresolved dataset quality issues.

• Frequent reassessments: Changes in use cases require repeated evaluations, even after data sharing has started.

• Frustration: Both requesters and providers struggle with inefficiencies.

• High resource costs: Data providers spend excessive time working with vendors to diagnose and resolve data quality problems.

Our Solution: DataSharingAssist

DataSharingAssist (DSA) is an AI-powered educational tool, enhanced with WOG policies and tools, designed to help public officers confidently manage inter-agency data requests. By providing data-driven recommendations on data privacy risks and data quality, DSA streamlines decision-making, making data sharing more efficient and stress-free—ultimately fostering openness and progress.

Key Features:

1. Interactive Data Quality & Privacy Assessment

• Visual tools to help users assess and identify privacy risks (e.g., overall score, high-risk columns, information uncertainty).

• Comprehensive data quality evaluation across multiple dimensions: accuracy, validity, consistency, integrity, completeness, and uniqueness (referenced from GovTech's DE-Playbook).

2. AI-Powered Conversational Chatbot

• A specialized assistant, augmented with WOG policies and tools, that provides guidance on data-sharing queries.

• Tailored recommendations based on assessed data quality and privacy risks.

3. Policy-Aligned Recommendations

• Automated suggestions to ensure compliance with government data protection policies.

4. WOG Tool Integration

• Smart recommendations for WOG tools, such as Cloak and Mirage, to help process and protect data for specific use cases.

5. Report Generation

• Automated generation of data privacy and quality reports, facilitating smoother decision-making and enabling seamless sharing with data requestors.

• Supports audit and compliance documentation, ensuring transparency and accountability.

Technical Implementation

DSA is built using advanced AI and visualization technologies to provide a seamless user experience:

• LLM Implementation with LangChain and OpenAI: DSA leverages a fine-tuned LLM for persona-based interactions, ensuring context-aware and user-specific responses.

• Retrieval-Augmented Generation (RAG): Enables DSA to retrieve relevant policy documents and knowledge, providing accurate, real-time guidance.

• Interactive Data Visualization: Built with Plotly, offering dynamic and intuitive visualizations to help users assess data quality and privacy risks effectively.

Development was streamlined with the coding assistant Windsurf, while deployment is currently underway using Airbase.

Impact and Outcomes

DSA delivers measurable improvements to the data-sharing process:

• Time Efficiency: Significantly reduces data request processing time, potentially shortening review cycles from months to minutes.

• Confident Decision-Making: Supports data-driven decisions with standardized metrics, improving confidence in assessments.

• Enhanced Risk Management: Identifies potential privacy risks and suggests mitigation strategies.

• Improved Data Quality: Assists data providers in detecting and addressing quality issues more efficiently.

• Streamlined Workflows: Simplifies data-sharing approvals with structured, automated workflows.

• Policy Compliance: Helps align with government data privacy protection policies through automated checks and recommendations.

Future Steps

Primarily based on insights from user research and the Feedback Bazaar, our roadmap focuses on further enhancing DSA's capabilities through the following improvements:

1. Enhanced chatbot with advanced reasoning capabilities

   • Improve the LLM’s ability to handle complex privacy scenarios and edge cases.
   • Implement a human-in-the-loop feedback mechanism for iterative model refinement.

2. Centralized discovery of public dataset repositories

   • Integrate a knowledge graph to link and index distributed public datasets.
   • Implement chatbot-assisted dataset discovery, enabling users to locate relevant datasets through contextual queries.

3. In-chatbot support for privacy measures application

   • Develop an agentic framework to identify the most suitable privacy tools (e.g., Cloak API) based on dataset characteristics.
   • Automate the application of anonymization, differential privacy, and synthetic data generation directly within the chatbot interface.

4. Expanded recommendation engine as a validator

   • Validate SCDF forms and other data-sharing formats to maintain consistent standards.
   • Extend recommendations to include dataset transformations and regulatory compliance checks.

5. Community-driven knowledge base

   • Build a collaborative repository of best practices, privacy guidelines, and real-world data-sharing scenarios.
   • Enable contributory updates from agencies to improve shared knowledge over time.

6. Collaboration between data requestors and data providers

   • Establish project-based workflows that support role-based access control, versioning, and secure data sharing.
   • Integrate notifications and document sharing within the platform so both parties can track requests, confirm data quality, and maintain compliance records.

7. User experience optimization

   • Continuously refine UI/UX for improved workflow efficiency and accessibility.
   • Incorporate user feedback and analytics to guide incremental improvements.

Conclusion

Our initial goal is to position DataSharingAssist as an educational tool for Whole-of-Government (WOG) adoption. It simplifies the complexities of data sharing, making it a seamless and efficient process that fosters openness and collaboration across government agencies. By leveraging AI-driven insights and automation, it enables public officers to make informed, secure, and policy-compliant data-sharing decisions.

The {build} 2025 program has provided an opportunity to address a critical public sector challenge, and we look forward to seeing how DataSharingAssist continues to evolve. As we expand its capabilities, our goal is to enhance data governance, streamline workflows, and strengthen trust in inter-agency data exchanges.

Meet The Team Members :)

Anshu Singh Research Engineer (GovTech, GTO)

Lara Chia Digital Business Analyst (MOM, GDT)

Brandon Lee Digital Business Analyst (Customs, GDT)

Jimmy Neo Senior Digital Services Manager (MOM, GDT)

Special thanks to Ghim Eng Yap, Director of Data Engineering Practice (including data privacy), for his invaluable support and input. He also serves as the business owner of this project.

Moment to remember: Team formation day a.k.a kick-off day ❤️

This project was developed as part of {build} 2025, a government innovation program. For more information, visit build.tech.gov.sg.