My Role: Development, animation, and effects
The ghost project is a space-based AR experience that’s part of an escape room. It shows the user a ghostly character who moves around the room that they’re in and provides hints to the escape room puzzles. It was initially a mobile AR app, but was eventually turned into a non-interactive screen-based AR piece.
The app is set up before each use by matching a pre-existing 3D scan of the room with the room itself. This is done by pointing the tablet camera around the room space and allowing the app to detect planes, so that the AR elements can remain steady and locked into place. The person doing this calibration (intended to be the escape room attendant) can then rotate and adjust the room scan mesh to match the room. The scan mesh then becomes invisible, but will still occlude the ghost and other AR elements, allowing them to look like they are moving behind or through objects in the room.
Then, while players are in the room, the app detects the presence of cards that the players find and scan. Each of the cards triggers a unique animation in which the ghost appears in a puff of smoke, moves around the room to give the player a hint about the puzzle they are trying to solve, and then vanishes into smoke again.
The rigged ghost model was provided to me for this project, and we recorded some mocap clips of an actress as a starting place for the ghost’s animations. My role was to develop the app, combine and modify animation clips (and create new ones as needed) and to create the realtime VFX for the smoke, the glow of the ghost, and the fluttering of her skirt as she moved. I put everything together in Unity engine and built it to work on an Android tablet.
The AR app did not end up being used — user testing showed that mobile AR wasn’t intuitive enough for first-time users. So I modified the project such that the animations could be played on a TV screen, overlaid on camera footage of the room. This way, users could see the ghost glide around the room with them without needing to interact with the device directly. This maintained the AR experience while simplifying the user experience.
For me, it meant modifying the virtual camera’s settings to match with the “security camera” type footage that was being used onscreen, rather than matching up to the tablet’s camera. After that I could just render out the animations as video clips with a dark background, which could then be layered over the live footage.
Despite not being used, the technical setup of the mobile app did inspire one of my other ongoing projects.
Created by Sprocketship for Mental Trap Escape Room Games.