How to distribute a database among microservices

In my earlier days as a software developer – long before joining Dynatrace as a Developer Advocate – we shared a database between two projects the wrong way. This is about how one bad decision taught us two important lessons.

A couple years back I was working as lead developer of a team working on a Java web application for a client. Let’s call this “Project A.” We were building the web app at the customer’s site and in addition to us there were a couple other teams working on related projects. As we all knew each other well from collaborating on earlier projects, we exchanged software architecture ideas between teams on a regular basis.

One day a new project (Project B) ramped up. This project was to be implemented by one of the other teams. In my mind’s eye I can still see us standing there; we were seen-it-all-done-it-all-know-it-all-wannabes making plans to share our oh-so-generic authentication-database schema for users, roles, and rights from Project A with Project B. After all, both projects were internal web apps serving the same userbase. Oh, yeah; I forgot to mention that there was no central user database for these applications—every new project was started from scratch. We figured we were not only saving days of development time by sharing existing infrastructure and know-how, we were also sparing the folks in Customer Support plenty of headaches because they wouldn’t have to deal with separate user directories.

We set things up so that the second project could access our database tables using a separate database user account because, you know, we didn’t want to make a big mess.

All went well, for a while…

Storing users, roles, and permissions isn’t rocket science after all. After about a year, plans for a new version of Project A arrived. We were excited because we then had the chance to keep what was good in Project A while improving the things that didn’t work so well. We even improved some parts of Project A that already worked well—among these, the database schema of the tables for users, roles, and privileges. To be honest, we weren’t even thinking about Project B.

Of course, Project B soon stopped working. The mistake we made was that we granted Project B direct access to the database. Not just by today’s standards but even back then, the right decision would have been to create a separate authentication service and to share a common API, but not database access.

But there’s more…

So, yes, we made a significant architectural error, but there was another issue that came into play. Ironically, Project B was a total non-starter. Despite having been implemented according to spec, with excellent test coverage and code quality, Project B wasn’t used much by the department that requested it. The project became stuck in a “friendly user” purgatory stage, ready for take off, but never formally launched. So it took about two weeks before somebody noticed that it had stopped working.

By the time the first unlucky user reported the problem, our devs were off working on other projects. As a first step towards root cause analysis we checked the error logs in an attempt to find out what the problem was. In hindsight, not having a sophisticated monitoring solution in place that would have automatically detected application problems was an equally poor decision on our part.

While this might be a funny annecdote, I really hope you learn a lesson from it. Make sure you access databases via a stable API as it turns that plain old database into a service and makes it shareable much easier later. And always make sure you monitor your apps and services properly. Environments built around APIs will keep your infrastructure dynamic in the long term. And monitoring makes sure that you keep the increased complexity under control.

I hope you enjoyed this little anecdote from back in the day, back when the first step of the deployment pipeline was to open the File menu and select Export as .war… in the Eclipse IDE.