Beware: rabbit holes
A few weeks ago - no lets be honest, a few months ago - we bought a massive bulk lot of second hand Lego from FB marketplace. We know for a fact there's a number of sets in there, so the plan was to sort them out and see what we could build. We've done this before, but this was the biggest lot we've had so far and we massively underestimated the amount of effort this would entail...
This evening I've been trying to pick out a specific Minecraft set and I was doing OK, til I started coming across a few missing pieces. Just the odd brick here and there, but before you build the set you have no way of knowing which pieces are there for decoration (so not completely necessary) and which are structural. So I figured I needed to keep track somehow of what was missing so I could either find an alternative or buy replacements from BrickLink.
This is where the madness began.
It started with a piece of paper I noted down on as I was going, which was fine for there and then in the moment but Lego and BrickLink don't always name their pieces, or the colours of the pieces, in a sensible way, so it was very hard to see at a glance what each note was actually about. And of course, analogue note taking has many flaws - it is easily lost, it gets super messy as you find surprise bits, writing is hard...
The next logical step was a spreadsheet. Well, a shared Google Sheet because why would it be anything else. It means we can both update things simultaneously, I could add some conditional formatting and functions and things so it was obvious what we were working on etc. But you have to get the initial parts list into the spready and copy/paste from BrickLink doesn't format well.
So obviously I wrote a little scraper for the site. I did look at their API but I had to sign up as a seller and I don't want to do that so I ended up with a combination of Beautiful Soup and Playwright (to get around the scraper-prevention) and saving the data as a CSV with the part images alongside. But when I opened that in Sheets it looked a bit shit, no pictures, messy data... Clearly I could do a better job if I had more control of the output view?
Now I have a local service with a tiny front end, so I can give it a set number and it goes off in the background and fetches the info from BrickLink and saves it as a local file. Then I have some endpoints that allow me to update the data with the quantity of each piece I have, again via an ugly front end.
Which is fine but what if we're doing it together? The thing about Google Sheets is that multiple people can work in one file simultaneously. So we need to host it somewhere centrally and have some live updating built in somehow. My little API has now been containerised and migrated to my home server. Websockets has been added so we get live updates when either of us changes anything, and it's now got a mobile-first front end because we'll mostly be doing this sitting on the sofa with our phones or ipads. I've also added a "missing pieces" export that I can use to order the bits we need from BrickLink.
It's still saving the data in json files though, because a database would be going too far...