I recently launched my latest side project, Dome. It's a tool that enables engineers to quickly and easily set up automated backups of files. As is typical with most of what I build, the initial idea for it came from a problem I faced.

Background

I have a thing for Telegram bots and have built a fair few of them, either as stand alone bots or as helpers to other side projects of mine (FactsonFactsbot, Yeetbot - Yeet, Mauibot - Maui). In building these bots, a factor I had to take into consideration was latency and speed of response. When you are chatting with a bot, you expect quick, snappy responses. A bot always taking a while to respond does not make for the best user experience. As a result, I made the decision to utilize the filesystem for data storage. The bots could quickly and easily access the filesystem for whatever information they needed which would mean two things

• For the standalone bots, it meant that there was no need for me store anything in an external API. Everything the bot needed could be accessed from the filesystem.
• For the helper bots, the number of times it had to query the external API could be kept to an absolute minimum.

This setup worked absolutely fine but over time I found myself wanting the security of backups for these file storage mechanisms I had implemented. At the very least, to be able to have them backed up externally to whatever server I had the bots running on. Just like how you have automated database backups as fail safes in worst case scenarios of data loss.

To implement these backups, I could just have easily written a simple script that would payload the different files at intervals to say an S3 bucket or cloudinary folder but I did not want to deal with the hassle of file management. As backups were being made, I needed non relevant backups to be deleted. The only backups I was really concerned about were the most recent ones. A backup made 3 months ago was of really no value to me. Also, I wanted to be notified anytime backups were successfully made or if for some reason, the backup failed. This is a lot for a simple script to do and would require me to have to setup a bunch of different things. With the bots, I only wanted to concern myself with the cron/job/automated mechanism that would payload the files for backup. The rest could be handled externally. As a result, I decided to build Dome to handle the "rest". Plus, in thinking about it, I realized that my use case was not the only possible one. For example, an application that needs to generate daily reports about whatever particular processes or events can easily leverage Dome to do so without needing to setup any external storage infrastructure or write any code to implement notifications.

How It Works

Users signup to the platform and are then prompted to create services. Services represent the applications whose files are to be stored. Under services, users create resources which represent the different files to be backed up. Each resource gets a unique Dome URL in the format https://{resource-uuid}.usedo.me for example https://3bb2e8c2-b07f-47c1-sdf1b-33h59de1f9ff.usedo.me. These files can then be backedup by making POST requests to their respective unique URLs. Ideally, with these requests, I would have really liked that the request body contained a single parameter - backup representing the file to be stored. The format of the file, whether JSON or txt or pdf could then be automatically inferred but I ran into several issues implementing that. As a result, along with the backup, the request body requires format representing the format of the file to be backed up.

Dome also offers several security features and mechanisms to provide layers of protection for backup requests. Users can enforce authentication for these requests in which case they would require an Authorization header of Bearer-{api_key}. API keys can be generated in the Dome dashboard. IP whitelists can also be setup to restrict access to particular IP addresses. For notifications, Dome offers three options. Users can be notified when

• backup requests are successfully made.
• backup requests are attempted with wrong API Keys
• backup requests are attempted from non whitelisted IP addresses

For now, these notifications are only available via email. I have plans to possibly build out Telegram and Slack integrations to enable notifications via these channels. I was tempted to build them first before launching but I realized that if I waited to do everything I wanted to do, I would probably never be done. Finally, Dome also offers three different duration options for backups. Backups for different files can be stored for 1 week, 1 month or 3 months. Automated deletions of backups older than the specified duration option occur when new backups are created.

Tech Stack

Dome is written entirely in typescript. There are two different backend services, Zilch written in Nest JS which is the major, monolithic backend which is responsible for every operation on the backend apart from recording backups. The smaller, microservice, Duran is written in Express. Its purpose is to record backups for the different resources/files. When requests are made to https://{resource-uuid}.usedo.me it is the service that picks up the requests. It was quite cool figuring out how to configure an Nginx webserver to respond to wildcard subdomains to implement that. Duran was initially written in Flask because I was looking for an excuse to write Python but I wasn't really satisfied with the developer experience of Flask's integration with MongoDB. I found myself having to make extra queries a lot of the time to fetch information that I could fetch once with Node. So I ended up re-writing Duran in Express. Much better developer experience like I said previously plus, why not just have everything in typescript. The frontends (Orion, Gello) are written in Next JS and React respectively. Orion is the application's landing page while Gello is the dashboard frontend from which users can configure and create all their services and resources. For styling, I made use of Chakra UI. I've wanted to learn it for the longest time, just needed to find a proper project to build with it. Which I guess I found with Dome. CI/CD is implemented with Github Actions with each of the frontends and backends running in Docker containers behind an Nginx reverse proxy via DigitalOcean.

Conclusion

Overall, I had quite a nice time building Dome and picked up quite a few new tips and tricks. I find it quite powerful facing problems and being able to build solutions to them. Like I said before I have additional plans to build out integrations (Slack, Telegram) through which backup notifications can be received. I also have plans for a playground in which users can make simulated backup requests to see the functionality without necessarily needing to signup. Or even if they are signed up, its would be a valuable tool to see how the platform works. I just need to tick off my plans to learn Vue and Fast API and get my hands dirty with a few projects before heading back to build that. Anyways, here's the link to Dome's github repo if anyone wants to poke through the code.