The next time you clean up your LEGO collection, you’re going to wish you had this machine at home to do the dirty work. Built pretty much entirely with LEGO pieces (plus a Raspberry Pi and some motors), this thing is able to sort virtually any LEGO piece that comes down its conveyer belts, thanks to artificial intelligence.
It uses a neural network—or a set of algorithms that recognize patterns, similar to the human brain—to match the real-world LEGO pieces with 3D images of the pieces that the machine has been fed during training.
The machine, which was built by YouTuber Daniel West, isn’t the first of its kind—though it looks to be the most effective. There are plenty of other contraptions on YouTube showing crazy machines that others have built to sort LEGO bricks, like machines that sort LEGO axles, specifically, and others that spin plastic cups around to catch parts.
West said that his universal LEGO sorter was inspired by two of these previous creations, including a 2011 model built by YouTuber Akiyuki Brick Channel, which was the first of these LEGO machines to sort a large number of pieces, and a 2017 machine that was the first to use AI to sort LEGO.
For over two years, West has worked on his LEGO sorting machine, and it can now recognize over 3,000 parts–the entire collection. The machine can even recognize parts that it hasn’t seen before thanks to AI. Using six LEGO motors and nine servo motors, West’s machine can sort about one brick every two seconds.
The key to this process is the machine’s so-called “capture unit,” which West details in a blog post on Medium. This is where the camera he installed takes footage of the LEGO parts under a light. The images are processed by the Raspberry Pi computer connected to the machine. Then, they’re sent wirelessly to a nearby computer that contains the neural network. That computer can use the neural network to analyze the image and send an output—what the piece is—back to the machine for sorting.
To do this, West built a convolutional neural network: a deep learning algorithm that takes an image input, assigns importance to it, and then classifies characteristics from one another. In a second YouTube video, West details how the AI side of the operation works. In short, the task is to take an input image and then make a prediction about what it is. To do that, you need to build up connections in the neural network; that is, you must use labelled data to show the machine different parts and what they are.
Since it’s really difficult to manually label images of LEGO, West used what he called “synthetic data.” He used a database of 3D LEGO parts to speed up the process. The problem? These fake images have subtle differences from real images in lighting, shadow, and texture that make it hard for the neural network to discern LEGO parts. This is called the sim-to-real problem.
To get past it, he relied on a concept called “domain randomization,” which essentially throws out the idea of trying to match the simulated 3D images perfectly to the real images of the LEGO parts generated by the camera in the capture unit of the machine. Instead, you can expand the possibilities for the types of images that are produced, resulting in more random images of LEGO parts with more variance in the color, shadow, and texture. Those images are fed to the neural network to train the connections in its “brain” better. To fine-tune predictions, West conducted a few more learning trials with real images. Voila, universal LEGO sorter.
While West is considering publishing the AI source code to make it open source, he’s keeping the mechanical designs for the machine close to the vest. He hopes to eventually publish an academic paper on the research behind his LEGO sorting machine. And if we can’t someday build this thing, we just hope we can at least buy it.