WOVEN REALITIES: DREAMCATCHER CONTROLLER
EXPLORING ALTERNATIVE INTERACTION METHODS – MERGING ORGANIC DESIGN WITH TECHNOLOGY
Developed in collaboration with Alejandro Rosado
Guidance and feedback provided by Daniel Overholt
Hello adventurers!
Dreamcast 2050 is an interactive spatial prototype, that features a surreal environment, a foggy open field infused with forest elements. The 270° multi-projection setup simulates a sense of space, surrounding the user with an atmospheric world.
1. DESIGN
Following a dream theme, a custom “Dreamcatcher” controller demo was designed to enhance Dreamcast 2050. Our goal with Alejandro was to explore alternative interaction methods beyond traditional controllers, keyboards, or other commercial devices. Inspired by folklore, organic shapes, and materials, the Dreamcatcher merges technological with natural elements with the aim of creating a more consistent and novel interaction experience. The body was created from tree branches, twine and was embedded with a gyroscope sensor. It symbolizes the connection between the virtual and physical spaces, combining the organic aesthetics of a traditional dreamcatcher with modern sensor technology while aligning with the dream theme.
2. DEVELOPMENT
The custom “dreamcatcher” controller was developed using an ESP32 WROVER module with gyroscope functionality. Gyroscope data was captured and processed using the Arduino IDE. To enable the ESP32 functionality, the ESP32 Board support package and the following libraries were installed in the Arduino IDE: MPU9250_asukiaaa, OSC, and Madgwick.
[ Controller connection diagram: The wireless communication path between the custom dreamcatcher controller and the PC system ]
A final, working firmware was uploaded to the ESP32 module, enabling it to transmit gyroscope data via Wi-Fi through a local network to the PC running the UE5 build. Sensor data was sent using the OSC and UDP protocols and parsed by a custom message processing program built using the VVVV platform. VVVV acted as a mediator between the ESP32 and UE5, creating a bridge for integration. This workaround was necessary as UE5 was unable to directly receive OSC data from the sensor due to currently unknown reasons.
[ Handmade Dreamcatcher Controller featuring an ESP32 WROVER sensor ]
[ VVVV implementation diagram: the OSC message handling system, both the receiving and transmission paths to UE5 ]
Within UE5, The gyroscope data was used to control the revealer’s position in the virtual space, allowing users to uncover hidden objects. The OSC plugin was installed, and an OSC Receiver Component was attached to the “Revealer” Actor. A server was created within the OSC Receiver to listen for messages from the local IP address. When gyroscope data (rotation values) was received, the specific address path was identified and processed.
However, the incoming data exhibited constant minor fluctuations, causing jittery behavior in the controller. To address this, a small deadzone threshold was applied, ensuring that minor changes below the threshold were ignored, allowing for a smoother interaction with the system. For values exceeding the threshold, the data was normalized using Clamp and mapped to control the X and Y axes of the Revealer object.
[ UE5 OSC receiver blueprint: Signal processing and translation into revealer movement, including jitter reduction and value clamping. Blueprints can be found here: OSC Receiver Component, Revealer Blueprint]
3. CHALLENGES
The development of the “dreamcatcher” controller was an ambitious project from the beginning. Despite understanding the difficulties involved, I was driven by curiosity to push the boundaries of my knowledge and explore the potential of creating a unique, alternative interaction device.
One of the initial challenges was identifying the right sensor model, understanding its pin configuration, and finding a functional library. Many available libraries and hardware were either deprecated or poorly documented. Technologies in this field evolve rapidly, and methods or tools that worked even two years ago are often no longer viable. Another particularly challenging aspect was establishing a stable communication pipeline between the ESP32 sensor and UE5. Initially, messages were not being received by UE5, so a debugging intermediary (VVVV) was introduced. Through this process, data was successfully transmitted from the ESP32 sensor to VVVV and then relayed to UE5. Although indirect, this workaround resolved the issue.
4. CONCLUSION
The dreamcatcher controller, despite its challenges, is a personal milestone in exploring alternative interaction methods. While issues such as jittery gyroscope values and calibration problems persisted, the project succeeded in creating a functional prototype. The project was also a valuable learning opportunity in hardware and firmware development.
Future work on the dreamcatcher controller includes addressing gyroscope calibration issues and considering alternative sensor options to improve interaction fluidity. Additionally, polishing the controller’s physical design by adding more decorative elements that would enhance its tactile feedback.
Resources
DIY: Make a Dreamcatcher with Tree Branches https://www.youtube.com/watch?v=-TE0JfKmuGA&t
LilyGO. (2024). TTGO TAudio: Hardware Documentation https://github.com/LilyGO/TTGO-TAudio
Random Nerd Tutorials. (2024). Installing ESP32 Board in Arduino IDE 2 (Windows, Mac OS X, Linux) https://randomnerdtutorials.com/installing-esp32-arduino-ide-2-0/
MPU9250_asukiaaa: A Library to Read Values from MPU9250 https://github.com/asukiaaa/MPU9250_asukiaaa
OSC for Arduino https://github.com/CNMAT/OSC
Madgwick Library https://github.com/arduino-libraries/MadgwickAHRS
OSC Plugin Overview for Unreal Engine https://dev.epicgames.com/documentation/en-us/unreal-engine/osc-plugin-overview-for-unreal-engine
Receive OSC Messages within Blueprints – Unreal Engine 5 Tutorial https://www.youtube.com/watch?v=DI2F-1iwOHA
2 Way Communication with OSC and Unreal 5.1 https://www.youtube.com/watch?v=zsu1jbumTIk
Controlling Unreal with OSC https://www.youtube.com/watch?v=9CkKPCBys44