Yesterday when I sat down to do some work, disaster struck. I opened my personal data management system aka my text file database and it was gone. Gone!! Tragedy! Oh woe is me!

But let's back up. What exactly is a text file database, why was losing it a disaster, and what exactly happened?

It all started yesterday morning....

So it Begins

I'm a Mac user. Try not to let that bias you; I know Apple hate is strong among a lot of people. Yesterday Apple released their annual update to the OS, macOS 14 Sonoma. Many of us feel that a yearly upgrade is way too aggressive, but Tim Cook is not a software guy, he's a numbers guy, and we all know what happens when a numbers guy takes over a company (see Microsoft when Ballmer took over). But I digress.

The update seemed to go flawlessly. Every year when Apple gives the new update it gets easier than the year before, and this year was the easiest yet. It only took about ten minutes and two reboots, and boom—new operating system. I suppose part of that speed may be that this year's update is focusing more on bug fixes and under-the-hood improvements, so there are fewer new features than usual.

I delayed my work while I played a little, looking at some of the new features and some of the other improvements. All in all it seems like a very nice improvement. Because of this work delay, I didn't notice the disaster.

Personal Data Management System: Zettelkasten

Later when I finally got to my work I opened an app called nvALT. This is the single most important thing I use and is critical to my work. The app points to a directory full of txt files, it allows you to easily and quickly search all the files, and allows one file to link to another, Wiki style. This is huge and is key to my work. This app enables essentially a personal Wiki or, as I call it, a text file database.

When you read a post from me full of dates and information about Japanese history or poetry or old systems of government, etc, you might wonder where I'm pulling all that info from. While the general info is in my head, most of the specific details are not. But I'm not running to my bookcase constantly to find this info. Instead I am referring to my personal data management system for the details.

These days this is not a strange idea. A few years ago someone on the internet rediscovered a personal knowledge management system originally created by Conrad Gessner in 1516 and then refined by Antonin Sertillanges in the 1920s called a Zettelkasten. Sertillanges's refinement used a card catalog system to easily link cards together and easily record any information one wants. Later in the 1940s a historian named Hans Blumenberg compiled more than 30,000 cards in his Zettelkasten and used this for all his work, which he used to basically know everything, and made use of it to write a number of academic books. There were a number of other people who used very similar ideas at the time. A sociologist named Niklas Luhmann one upped that; his Zettelkasten has 90,000+ cards, enabling him to write over 50 books and 500+ scholarly papers.

While the Zettelkasten has always been well-known in some research and academic circles, it wasn't widely known. Since being rediscovered by the internet, however, this Zettelkasten idea has become fairly popular in personal development circles.

^{How To Use the paper version of the Zettelkasten. From Wikipeda}

I had never heard of Zettelkasten and had independently created the same idea using nvALT; or, perhaps I had borrowed the Wiki idea and modified it into my own system that ended up mimicking the Zettelkasten. At any rate, I use my text file database for all my research, both personal and professional. Everything I study, everything I learn, everything goes into a note (a text file) and it is most likely linked to another note, or another note is linked to it. While I do not have nearly as many cards as Luhmann, I am approaching 10k. I would probably have more than 10k if I followed Luhmann's philosophy and had only one idea per card. Since learning about the Zettelkasten and Luhmann's ideas on its use, I try to stick with one idea per card, but for many years I usually included multiple ideas per card, which probably makes the system less effective.

Let me give you an example of how it works for me.

In my haiku research I have a card or note (a text file) with a list of all haiku poets I know, like, or want to hold some research about. Each of the names in this note are then linked to a note with info about that poet. The poet note is then linked to a note with links to each of their haiku that I have recorded, translated, or gathered info about. Any one of these haiku notes might then be linked to more notes, for example a note about the kigo (season word) used in the haiku or a diary entry that surrounded this haiku. At the bottom of every note I carefully record where I found the info and how trustworthy the source is (original sources or books by scholars are more trustworthy than magazine articles which are much more trustworthy than web pages).

You can see the result of this is a complex web of notes linking to each other, not unlike the internet itself.

An App Neglected

The app I use to do all of this, nvALT, hasn't seen development in a long time. It was forked from an open source project around 2011, and that is when I started using it. But active development stopped in 2013. After that it received bug fix updates until 2017, but nothing since then. The developer started coding a replacement for this app in 2013 but it hasn't see a release yet. I am on the beta for it and the developed has been active, giving new versions weekly or even daily for the past several years, but these releases are still nowhere near release quality and are still not to the level of nvALT.

It's an understandable situation. The developer never charged any money for nvALT so there is nothing to fund development on the new app, meaning it has remained something he can only do in his free time. I and many others would gladly pay, but at this point not enough people would pay without a new product to support him, putting him in a tricky situation.

At any rate, this is a problem for me. Because of this, every time there is a new OS release I cross my fingers and hope nvALT will continue to work.

Disaster!

Back to the story. So after playing with some of the new features of OS 14 for awhile, I decided to get down to work. I had taught some students earlier in the day so I was going to add notes about how they did and what we covered in class to the cards I have for them in my text file database.

I opened nvALT and.... it was blank.

Well, shit, I thought. nvALT has finally stopped working. Now what the ... am I going to do?

The first thing I tried was I loaded up the latest nightly release for the nvALT replacement (called nvUltra). I thought even if it isn't as good as nvALT, I might as well get used to it. ...It was also blank. Crap! Did the new OS also break nvUltra??

I decided to go directly to the directory on my computer containing all the individual text files of the text file database and just open the files I need directly. The file search of Finder (the macOS file explorer) is not as good or as fast as nvALT, but it does work in a pinch.

...the directory was also blank!

Well, at least now I knew what the problem was.

Lucky!

Evidently all the files in the directory has been deleted when I upgraded the OS. Thinking back, I had probably had nvALT open when I upgraded: my guess is that something in the upgrade triggered the delete all action in nvALT and then a reboot dismissed the "are you sure you want to delete these files" dialog window before I could see it, then a bug in nvALT allowed it to go ahead and delete them all even without my explicit permission at the confirmation dialog window. That's only a guess. I have no idea.

I am not the type to panic, but I was a little worried. Ten years of research down the drain!

Luckily I was able to find what I assume to be most or hopefully all of the files sitting in my trash. This is very lucky because I have my trash set to automatically delete all the contents after reaching a certain size. Phew! Lucky!

I restored the files and then immediately zipped up a copy of the directory and put in on my backup drive. Looking on the backup drive, I saw that the last time I backed it up this way was July of last year.

Safeguarding the Future

Now that my text file database was restored, I set out to ensure I backed it up on a more regular basis. I used to clone my hard drive daily and I should go back to this practice, but until I do so I want a solution to at least back up this most important aspect of my life.

In the past I might have automated this using AppleScript, but a year or two ago Mac added an automation app called Shortcuts that makes programming little jobs like this even easier. Here is my resulting script:

This script takes the entire folder of my text file database, zips it up, and names the zip DATE text file database bak where DATE is today's date.

Using a different app, I set this script to run everyday at a certain time. Then I have the OS set to automatically move the resulting backup file to my backup drive

Again, using a cloning indexed backup program would definitely be a better solution, especially if I ever need to compare a current version of a file to a previous one. My script is not a good solution. But it is a solution for now and anything is better than nothing.

A Lesson

The moral of this story is backup your files.

If I hadn't been so lucky to find all my files sitting in the trash, I would have been out a year of work, having to revert to my last backup which was from over a year ago. I know to backup, but like an idiot who says he's only driving a block to the store so he doesn't need to put his seatbelt on, I had been stupidly not actually backing up, holding onto the idea that it won't happen to me. Well, it almost did happen to me! And it might to you too. Back up your files, at least the most critical ones.

(misc: all photos by me)

David LaSpina is an American photographer and translator lost in Japan, trying to capture the beauty of this country one photo at a time and searching for the perfect haiku.