Gesture.js: A Cloud-Deployable Framework for Building Embodied Experiences

Emerging body movement detection and gesture recognition software have opened a gateway of possibilities to make technology more intuitive, engaging, and accessible for people. A vast areaof natural user interfaces is leveraging body motion tracking and gesture recognition technologies and a human’s readily expressive body to extend interactions with software beyond mouse clicks and scrolls. However, these interfaces have been limited by hardware and software expenses, high development time and costs, and learning curves. This paper explores different approaches to providing both software developers and designers with easier ways to incorporate computer vision-based body and gesture detection solutions into the development of embodied experiences without suppressing creativity. Gesture.js is a JavaScript framework as a service (FaaS) that is both a thin library on top of the Document Object Model (DOM) consisting of a collection of tools for developing embodied-enabled applications on the web and a landmark computation and processing application programming interface. It wraps MediaPipe, an open-source collection of machine-learning solutions that perform inference over arbitrary sensory data, and additional landmark processing frameworks such as KalidoKit, a 3D model rigging solution, and ports the necessary information through either an object-oriented or an API-oriented implementation. It also comes with its web-based graphical interface for easy connection between Gesture.js and other application clients with little to no JavaScript code. This thesis also details a collection of example applications that demonstrate the usability, capacity, and potential of this framework.