Welcome to the Mandala project sketchbook. This is where I've collected some interactive sketches for exploring the systems that go into the project.
Current status is that we've got the pose estimation, body model, and two promising Mandala models working. The nature model is TBD (and might end up being dropped for time).
The main goal at this point is nailing down a direction for the mandala and figuring out how to integrate the pre-baked animations.
Conceptually, we proceed through three stages: input, modelling and rendering. The input stage takes in camera frames and processes them into body pose information. The modelling stage uses the input to manipulate models that represent the internal logic and structure of the system. Finally, the rendering stage takes those models, turns them into pixels and sends them to the projector.
This is a demo of the whole system working together. For now, it only shows the plane wave model along with pre-recorded movement.
The model is controlled by a few (currently non-configurable) paramters from the body model. During static poses, the limb positions are mapped to the directions of the plane waves. During dynamic movement, the arms are mapped to phase shifts, rotation and zooming.
The first step of the system is the camera input and pose estimation. This is likely an RGB+Depth camera (eg Kinect) but it also works okay with a regular camera.
The reason we may need a depth camera is because we're projecting on/around the body we're recording. Also, if our framerate for pose estimation is slow, we may be able to make it faster/more responsive by using the depth pixel movement to distort the skeleton.
This demo uses the ViTPose model for pose estimation, but there are a few other options, so this will likely be tweaked based on real-world performance. Don't worry if it's a bit glitchy on the pre-recorded video – there isn't much light on the dancer, so the model gets a bit confused about which direction they're facing.
The body model is responsible for integrating the pose frames we get from the input, maintaining a coherent skeleton, and providing abstract information based on how that skeleton moves.
The two main kinds of information we want are shapes & gestures.
Shapes are created by briefly holding static positions. We look for interesting geometry and symmetries created by the limbs and provide shapes that construct the mandala.
Gestures are body-referenced movements with properties like rotation, divergence (inward/outward), direction and speed. These are used to manipulate the mandala. The idea is ultimately to have the most extreme gestures ultimately release it to our nature model.
Our mandala needs some essential properties. It has to represent some kind of rotational or reflective symmetry. It has to have intricate internal detail. It has to have a number of tuneable parameters that allow it to be smoothly varied, but it also has to be capable of dramatic transitions from one shape to another.
Ideally it also connects with the properties we get from the body model. ie, it can incorporate shape and movement in a way that creates a visible conceptual connection between the two.
We have a number of possible models, and our final mandala is likely to be a combination of a few of them, so this is a place to explore that.
(not uploaded yet)
The nature model is a way of creating foreground detail.
Sprites are how we display pre-rendered illustrations/animations. They are fairly simple, consisting of a frame sequence and a central coordinate for where the sprite should appear.
The one complex element is that sprites can have rules attached that make them move in certain ways and even generate new sprites. This is key to the sprites being able to display natural movement.