(Only works on Firefox, since it uses Audio Data API.)
The app is not most intuitive and has a tiny learning curve.
Please refer to more info section.
Video: http://www.youtube.com/watch?v=bl11T112Jn0
About
CodeDJ Live! is a web-based app for programmers to code up visuals in real-time as the music plays. With this app the programmer becomes a Visualizer DJ who produces stunning visuals at a party.
The app is inspired by Iñigo Quilez and his live code demos at SIGGRAPH 2012 Real-time Live event. Similar examples can be seen here and here.
Technical Overview
This app is essentially a WebGL GLSL live coder hooked up to audio spectrum analyzer. The user has access to the GLSL code that is producing the visuals, which can be compiled instantly and displayed.
Technical Details
Music Analyzer
Audio frequency analysis is done using Mozilla's Audio Data API and fast-Fourier transform functions, provided by [blank]'s signal processing library. The Audio Data API provides access to the framebuffer, which contains the decoded audio sample data. This data is fed into the fast-Fourier transform to extract spectrum frequency data. Frequency data has 1024 channels. This is way too many for the use of this app and needs to be reduced to a manageable and meaningful number. The higher half of the frequencies are barely noticeable, so they are ignored. The lower half are kept and averaged down to 8 channels total for the user to use.
Music Access
Audio Data API requires access to the audio framebuffer data, which is sensitive secured data access. A limitation to this is the "same origin policy", which means it cannot access audio data outside the same root website. This becomes a problem since users need to be able to use their own music for this app. A solution is to host the web app on a public directory on my personal Dropbox, and have users provide their music using their own Dropbox. This is done through the Dropbox javascript API. This works because now the web app and any user provided files are all within the same root website, which is dropbox.com.
Music Streaming
The user logs in through Dropbox API and specifies which folder to contains the music files. Only .ogg and .wav files are accepted, as determined by Mozilla Firefox. Once the user has selected a folder, no music files are downloaded or transferred. Instead, public URLs of the files are generated and stored within the app for that session. These URLs expire within 4 hours. The audio is then streamed per file as they are needed, using the URL. The app does not load more than one audio file at any time.
Visuals Shader
Visuals are generated using a fullscreen quad and a fragment shader. Audio frequency levels are passed into this shader as uniforms. The GLSL code generating the visuals is exposed through a large text editor, where the code can be edited and compiled in real-time to be displayed on the screen.
Postprocessing Shader
Finally, the visuals go through a post-processing shader which applies a series of custom effects to produce dance-club-like styles. This includes over-saturation, bloom, box-blur, and film grain.
No comments:
Post a Comment