This project explores how serial data written by environmental sensors can control and influence the appearance of digital media. The outcome of Serial Link was a wearable device that incorporates a ESP32, Gyroscope, ultrasonic sensor, and a power bank that can control digital media in touch designer through network access.
What is serial data?
Utilizing the serial, we can send across our data through sorting how it’s written in csv.
The microcontroller hosts a network in which it sends through the values picked up from it’s sensors through writing the data to the serial.
The data is split like such: X,Y,Z, Roll, Pitch, Depth.
What is controlled data?
Controlling the data and knowing how to handle it is a big part of the system. This allows us to create controlled interpreters for this data. One for feeding it into touch designer, and the other feeding it into retroarch.
What is Touchdesigner?
Touchdesigner is a node-based visual programming language for real-time interactive multimedia content, developed by the Toronto-based company Derivative. It’s been used by artists, programmers, creative coders, software designers, and performers to create performances, installations, and fixed media works.
Early tests – reading mouse pose data
As a simple first step for this project that aided in developing the logic and rule set for how to manipulate a simple spherical geometry was done by linking the mouse position to positional inputs in touch designer.
This small exercise gave us an understanding of how to scale high value numbers from the mouse input into more manageable dataset that would reduce randomness of values, creating a smoother flow of interaction with the geometry – at the time we didn’t know it, but we had unknowingly filtered and averaged out the serial read, which would greatly aid us in our next experiments when incorporating environmental sensors.
Ultrasonic sensor implementation
To ensure that the geometry would never get too big we set a min and max value range that the ultrasonic sensor would detect. This would then be input into touchdesigner where the raw serial data was remaped to a new range more manageable for manipulating scale.
Gyroscope sensor implementation
As with the ultrasonic sensor we set a min and max value range that the gyroscope would detect. This would then be input into touchdesigner where the raw serial data was remaped to a new range more manageable for manipulating scale.
Combined sensor integration
Since we were able to preform isolated tests on each sensor prior to combining sensors we knew the range and limits of so the result ended up working within the first few attempts
Testing sensors on 3D data
Sensor logic breakdown
Networking setup
In this setup, the ESP32 serves as an Access Point (Wi-Fi hotspot), enabling a laptop to establish a connection. The laptop then sends requests to the ESP32 for sensor data. Utilizing a TCP/Internet protocol, TouchDesigner, running on the laptop, communicates with the ESP32 to receive the requested data. This configuration facilitates wireless data transmission between the sensors connected to the ESP32 and TouchDesigner, streamlining the integration of sensor data into the creative process facilitated by TouchDesigner.
Schematic
Further steps
