A Singapore Government Agency Website
How to identify
Provisioning databases in one click
As a team made of developers/tinkerers, we recognised that databases were a common and major point of friction when developing applications especially when most applications require a way to persist data. We wanted to provide a solution that would allow GovTech engineers to quickly spin up a new database for development and expedite their workflows for iterating quickly.
This problem was also observed from repeated feedback shared by many Airbase users, where developers were always asking for a database service they could use with Airbase's platform for rapid application development.
Through conducting a survey, we also found that most developers wanted to use Postgres as their database, leading to our choice to specialise in provisioning Postgres databases instead of other databases, though this is open to change as the solution matures.
GovTech Engineers/Developers need to have a database hosted on GCC for prototyping and development work when new ideas or requirements are surfaced by agency or public officers.
But setting up a new GCC tenant takes a long time, processes to get infrastructure provisioned takes weeks or months, there are high coordination costs between teams, and setting up networking for databases and developer endpoints is challenging.
This results in frustrated and burned out engineers, slower software delivery for new ideas, drift between local and remote databases, and long time to market for new initiatives.
GovTech engineers need to provision a Postgres database when needing to persist their application data, but provisioning a secure database takes a long time and requires expertise, resulting in increased time required for application development.
We researched the market and explored various options, including:
Eventually, we decided to leverage existing commercial solutions (Neon) and created a backend that wraps it with our own authentication. This allows us to centrally manage created databases and use features locked behind Neon's paid plans without having to pay per database. It is a novel approach to time-sharing a single Neon subscription to validate our hypothesis around the need for faster provisioning of databases to enable developers to iterate faster.
We also created a CLI that works on Mac, Linux, and Windows machines that allows developers to easily provision databases from the command line.
This also makes it easy to provision databases from CI/CD pipelines, which may be useful for testing.
Reduction in time taken to create remote and shareable development databases from hours to seconds.
Overcome the friction of collaborative workflows for local databases (Postgres containers or Postgres app on Mac), reducing the time taken to share/troubleshoot data related issues within development teams.
Our solution delivers a working but limited solution to the identified problem. It is not yet suitable for production use as it does not adhere to IM8 or other government security or compliance standards.
This still makes it suitable and useful for prototyping and hackathons, as it is a quick and convenient way to provision Postgres databases.
It is also still suitable for temporary databases which may be useful for testing.
We are yet to expose Neon specific features like database branching as it would lock the solution into Neon, but in doing so we lose a lot of the value in using Neon.
There is a risk that Neon stops offering its SaaS solution, which would require us to migrate to a different underlying technology. We do have other potential solutions worth looking into, such as . This would also not be an issue if we self-host Neon instead of using their SaaS offering.
Going the self-hosted route also has a risk that Neon stops maintaining or supporting their open-source work. Doing so would force us to maintain it ourselves or move to a different solution.
Postgres may also fall out of favour with GovTech engineers, but we deem that to be unlikely within the next decade. Our solution is also not overly coupled to Postgres, so offering multiple database technologies is possible.
According to the , Postgres is both the most commonly used and the most desired relational Database.
The results showed that 277 (80.8%) of its 343 respondents have used Postgres in production.
The runner up, MySQL, is quite a bit behind with only 191 (55.7%) respondents, 31.0% less than Postgres.
Respondents also showed a strong preference for Postgres, with 225 (65.6%) of them selecting it for their ideal tech stack.
The runner up, DynamoDB, was only selected by 44 (12.9%) of respondents, 80.4% less than Postgres.
All of the top 5 ideal tech stacks selected Postgres as their database of choice.
We can consider or explore self-hosted Neon on GCC2/GCC+ to better adhere to our security and compliance standards. With control over the hosted infrastructure, this would allow us to use IP allow-lists and/or restrict usage to trusted networks.
We would still need to further our research on how engineers provision databases when setting up for new projects/ideas.
As part of the hackathon, we've not explored how teams are operating with databases when they are used in staging or production environments.
These could require understanding existing workflows for backups, restoration and deployment of application changes that involve database schema updates.
