LegoGPT turns text into full LEGO building instructions

Researchers from Carnegie Mellon University have created a neural network called LegoGPT, which can generate assembly diagrams for models from LEGO parts based on a simple text description. The developed system generates not only an image of the finished structure in PNG format, but also step-by-step text instructions, as well as a CAD file in .ldr format, suitable for use in specialized programs.

The model was trained on an extensive database of 47 thousand LEGO structures, each of which was modeled on the basis of 28 thousand three-dimensional objects from ShapeNetCore. To check the stability of the models, the mathematical optimizer Gurobi was used, and the description was created using GPT-4o. LegoGPT uses a modified version of Llama-3.2-1B-Instruct as a language base.

The neural network recognizes 21 categories of objects – from vehicles to furniture and musical instruments, but does not yet work outside of these groups. The project code has already been published in the public domain on GitHub and the Hugging Face platform.

The creators believe that LegoGPT can significantly simplify the creation of custom sets, simplify interaction with CAD environments, and also find application in the fields of education, design and industrial modeling.