Human-in-the-Loop: Interactive Systems
Faculty: Hamid Peiro + Aleksandra Kraeva
Hardware III is a cross-discipline project led by Hamid Peiro and Aleksandra Kraeva (Sasha); between MRAC and MAAI to explore the possibilities of interactive design within and around the boundaries of the worlds of robotics and AI. The project aims to better understand how the human body, decisions,and movements become the logic that drives a fabrication process. For our groups project we wanted to ask not just how may the machine react to what the human does but then; in vice-versa. How may the human react to the machines response. A bi-directional conversation between analogue and digital.
As an architectural school project we thought about what kind of space can be provocative to cause this reaction and looking at the world of Italo Calvino and Jorge Luis Borges began to explore memories as both an archive of experienced spaces but perhaps a future design tool for spaces also.
Discussing how to achieve this process we decided on a mixed reality output where AI would generate a virtual layer representative of the memories on top of an actual space the user would be in reality. This was simplified within our means for the project to: a LLM questionnaire and ESP32 sensor system received by Touch Designer as JPEG and CHOP data and then exported on top of photogrammetry generated space of our classroom via a Three JS output.
IAAC classroom 303
Metashape photogrammetry model
TouchDesigner (TD) output
System Architecture + LLM
This system architecture map shows a full breakdown of how the project works. We wanted to explore Speech to text as a non-intrusive way of asking the LLM questions when the user was immersed. Also as a group from varying cultures all over the globe: Ethiopia/ USA, British, Chinese, Bangladeshi and Taiwanese we were excited to see how the model could deal with non-anglophonic requests and produce culturally relevant answers.
This discussion of what imagery would be appropriate and what questions we should ask led to the generation of a reference library taking from the legendary film directors, whom all also discussed how to best represent the imagery in themes of memory and time.
Wong Kar-Wai
Hiroshi Sugimoto
Andrei Tarkovsky
These are three examples of the image generation this produced from the final LLM model.
Finite State Management + OpenCV / Sensors
As the aim for the project was for interactive design we heavily discussed how the UI/ UX could be maximised using sensors both as inputs to capture the user’s reactions effectively but not intrusively. Originally we had wished to use EEG headware which read electronic signals from the brain but unable to attain this used an ECG system instead with microphones, temperature along the heartbeat sensor. The idea being if the user reacted emotionally strong to the memory this would cause a reaction in the projected LLM output through Touch Designer. Alongside this we used OpenCV to confirm the prompts and state of the process.
TouchDesigner + Three.JS
When exporting to Three JS webpage we used a Node JS handshake to export a series of frames of the point cloud overlayed with the LLM ouput. However, an issue this faces is when the user manoeuvres in Three.JS they move in isolation and at different orientation to the TouchDesigner frames. To solve this we used a bounding box system of co-ordinates fed back through the handshake to the TD file. Therefore when the user moved in Three.JS this was fed back into the TD orientation and the two moved equivocally at the same time.
Summative Outcome
Here is an overview of all our system including the Image Bind into Stream Diffusion TD input. This is then output into a Three.JS webpage navigated using Whispr (S2T) and OpenCV hand gestures.
