Mahan Mehrvarz, Jerry de Vos, Aadjan Van Der Helm, Martin Havranek
Prompting Realities aims to create a tangible gateway to a future where LLMs mediate between people and products. The project leverages recent LLM developments to stretch the imagination, envisioning a scenario where âiterative ChatGPT-styleâ interactions move beyond screen-based outcomes and enter the realm of physical reality. This shift highlights many possibilities, including blurring the boundary between design and use, and empowering users with greater agency in the design and engineering of their everyday products.
Can AI upgrade your relationships with your products?Â
The application of LLMs in our everyday lives has gained momentum since the ChatGPT breakthrough of November 2022. While much attention has been focused on ways to engineer prompts for specific textual results, little emphasis has been placed on how such iterative prompt engineering techniques can be extended beyond the limitations of screens and into more tangible, physical realities. This shift can empower users, expanding the familiar concept of âend-user programmingâ to include any type of everyday artefact.Â
Consequently, Prompting Realities envisions an experiential scenario where AI distributes the authoritative role of engineers and designers as creators to end-users, opening up new opportunities to contest, interrupt, resist, or manipulate everyday products under unfair situationsânot through a system, but at the edge of usage.
The underlying pipeline operates with a simple yet powerful approach: providing a precise description of the prototype (functionality, appearance, etc.) to the large language model. This enables the model to understand the correspondence between computer variables and the effects they can have on reality, in relation to the prototype's functionality. There is little emphasis on specific hardware or software solutions used in the current prototypes, as the project primarily focuses on offering a new user experience rather than introducing a new AI-powered system.
The prototypes utilized a range of technologies, including the OpenAI Assistant API, Telegram Bot Interface, and TU Delft IDE Connected Interaction Kit, all of which can be replaced by similar alternatives.
Prompting Realities explores the intersection of AI and physical computing, expanding the application of large language models beyond text-based interactions into tangible, real-world contexts. By empowering users to interact with AI-driven prototypes through conversational interfaces, the project pushes classic notions of end-user programming, redistributing control and agency from engineers and designers to everyday users. This shift has the potential to democratize technology enabling resistance, interruption, and subversion opening new avenues for human-AI collaboration. Through experiential AI prototyping, the project provokes speculative reflection on AIâs role in shaping our physical and digital environments, encouraging deeper engagement with the evolving relationship between humans and machines.
