Wow.. theres a lot more to database backups than I thought

Nipunn Koorapati, one of the OG engineers here at Convex, sat me down at a whiteboard to explain why backing up a database is a lot harder than it looks. I went in thinking backups are pretty simple, you just loop over every table and copy the rows out right? Well turns out no, if your data is changing while you copy it you can end up with a backup of data that never actually existed at any point in time. We start from that naive version and work up through the document log, snapshots and the back and forth "dance" that sorts it out, then the new data sync endpoint, what OLAP actually is, and what's coming next for backups. I learnt a ton from this one, so grab yourself a lovely cup of tea and let's dive in. Backup and restore in the docs: https://docs.convex.dev/database/backup-restore The data sync endpoint: https://docs.convex.dev/deployment-api/data-sync Streaming export to Fivetran and friends: https://docs.convex.dev/production/integrations/streaming-import-export The Convex connector on Fivetran: https://fivetran.com/docs/connectors/databases/convex Nipunn's posts on Stack: https://stack.convex.dev/author/nipunn-koorapati My video where I crack open the SQLite file: https://www.youtube.com/watch?v=3d29eKJ2Vws My whiteboard session with Ian on time in Convex: https://www.youtube.com/watch?v=pHrxmSjxLg8 How do you back up your database at the moment? Let me know down in the comments. Chapters 0:00 Surely backups are simple? 1:46 Where do we start? 4:02 Where does this go wrong? 6:37 What does a database actually store? 9:35 How does the log help? 11:13 So, what do we do? 13:32 Any other issues to worry about? 16:09 What's the better algorithm? 21:36 How does it all fit together? 25:53 What's next for backups? 27:06 What even is OLAP? 38:20 Anything else about backups? Thanks to Nipunn for taking the time, and for putting up with all my dumb questions.