SwiftOnboard

SwiftOnboard Team members and divisions

Problem Statement

How Might We streamline the data onboarding process and improve collaboration, responsive to changes and improve knowledge sharing between data producing and consuming agencies to ensure timely and high-quality data for effective policy making?

Problem Formulation Process

Team carried out a user interview with our problem owner to understand the user’s pain point and challenges that changed our problem statement with better understanding of user's pain points.

Key Insights Gathered from the business owner Interview

(1) Challenges of current onboarding workflow

• The current workflow (preparation > development > UAT > operation) is time-consuming, taking weeks to onboard a single dataset.

(2) Legacy Data Issues Are a Persistent Problem

• Legacy data issues are often discovered only after datasets go live, indicating a lack of proactive quality checks during the onboarding process.
• Data-producing agencies de-prioritize fixing these issues, shifting the burden to consuming agencies, which delays policy making.
• Data-producing agencies face frustration due to repeated requests for the same dataset from multiple parties, highlighting inefficiencies in data sharing and collaboration.
• Consuming agencies lack domain knowledge, making it difficult to work effectively with development teams and adapt datasets for their needs.

(3) Minor Data Errors Cause Major Disruptions

• Even a 1% error in data is considered critical and requires immediate resolution, disrupting ongoing operations and diverting resources from other priorities.

(4) Data Management Knowledge & Skill Challenges

• Data providers lack capacity to handle requests, leading requesters to make decisions on data usage and behaviour.
• Data Request POs are expected to become data experts quickly (within 1 to 2 weeks), causing rework due to their limitations
• Data Request POs need project management skills to overcome knowledge gaps
• Lack of domain expertise leads to rework and inefficiencies, the knowledge gaps result in firefighting situations after dataset go live

Proposed Solution Details

Vision for SwiftOnboard We envision Data Onboarding Platform (SwiftOnboard) end goal to be a shared, cloud-based ecosystem where data consuming agencies can work with government data seamlessly. This can be achieved through a centralized knowledge hub for domain-specific guidance.

Ideation

• Features that allow officers independently prepare and customize datasets will result in faster and efficient data onboarding because the current workflow (preparation > development > UAT > operation) is time-consuming, taking weeks to onboard a single dataset.
• Features that allow officers to make changes in response to new things discovered when working with dataset will result in officers responding to urgent and immediate operation need to change the data logic and mitigate legacy data issues efficiently because legacy data issues are uncovered after dataset goes live is disruptive to operation and affects data quality which requires often urgent and immediate data and logic rectification steps.
• Features that establish baseline dataset with pre-defined rules to enable rapid onboarding of common dataset used by agencies will result in improve data domain knowledge gap and speed up data onboarding to meet policy deadline because data consuming PO are forced to be the data guru within 1 to 2 weeks self-learning even though they are not the data domain expert. PO roles require high project management skills to bridge the knowledge gap. However they are not the domain expert and will cause rework due to their limitations of not knowing what they don’t know.
• Features that monitor & detect errors with capability to automate resolution to data error issues will result in officers responding to urgent and immediate operation need to change the data logic and mitigate legacy data issues efficiently because data quality is rated 9/10. However, when the 1/10 data quality issue happens, it will cause an urgent and immediate snowball effect for recovery. Ad Hoc change to patch existing data, re-ingestion of data file, modify pipeline logic.

Impact and Outcome Analysis

Future Plans