WebTube

Here is a long overdue post that I finally got around to write now that school is over! Over the weekend of January 18-20th, I attended the Spring 2013 PennApps Hackathon, the largest student-run hackathon on the east coast, organized by no other than the University of Pennsylvania. Since this was to be my last PennApps Hackathon (this being my 5th time), I decided to have fun over the weekend, hang out with people, and do something fun.

I made WebTube, a fun little web app that lets you browse the internet through the comfort of an old-school CRT TV.

The reception turned out much better than I had expected. I came into the hackathon only to have some fun and not expecting anything. In the end, WebTube won 3rd place overall and Audience Choice Award!

Demo video: http://www.youtube.com/watch?v=8ZUCyN6yvps
Live code: http://iamnop.com/webtube/ (must enable CSS Shaders flag in Chrome)

Technical Details

The core component of this web app is CSS Shaders, which allows access to programmable vertex and fragment shaders for HTML DOM elements. CSS Shaders are an extension to CSS Filters, which are a set of image filters available for CSS, such as grayscale, sepia, brightness/contrast, hue/saturation, etc. CSS Shaders are essentially custom filters that can be controlled through vertex and fragment shaders. This allows for a much greater freedom and very interesting CSS animations and CSS transitions. (See CSS FilterLab for live examples.) Please see related links below for more information related to CSS Shaders.

With understanding of CSS Shaders, WebTube is a very simple app. It has one main DOM for the screen, which allows for different modes, including web and SSH terminal. Multiple CSS transition rules are applied to the screen using CSS Shaders. The vertex shader warps the screen to produce the curvature and the fragment shader applies TV-like effects on the screen.

Links

http://www.adobe.com/devnet/html5/articles/css-shaders.html
http://html.adobe.com/webplatform/graphics/customfilters/cssfilterlab/
http://alteredqualia.com/css-shaders/article/
https://dvcs.w3.org/hg/FXTF/raw-file/tip/filters/index.html#shading-language

WebGL Volumetric Renderer

 

Live code: iamnop.com/volumetric
(Does not run Windows. Unknown bug with shader compiling.)

Video: http://www.youtube.com/watch?v=VPhnwOpmUqY

GitHub: https://github.com/nopjia/WebGL-Volumetric

As a small side project, I implemented volumetric ray casting in WebGL. The idea is to ray cast a 3D texture, which should be very fast since OpenGL texture lookups are highly optimized; however, this is not doable since 3D textures are not allowed in WebGL.

During SIGGRAPH 2012, I met Luis Kabongo from VICOMTech, who showed me their implementation of a WebGL volume renderer, which they use for medical imaging. They made it possible by, instead of using a 3D texture, using a 2D texture atlas, which is made up of 2D texture slices. This is a very simple solution. I immediate took this idea and implemented my own version.

VICOMTech's volume renderer is for simply viewing 3D datasets, so they implemented it using alpha compositing with no lighting.

I took it a step further for my own renderer and implemented physically-based volume rendering. The lighting model is based on light transmittance. I introduced multiple lights into the scene and use exponential fall-off to calculate transmittance.

As a result, the renderer features physically-based lighting, volumetric shadows, and support for multiple lights. The results turned out very nicely.

CodeDJ Live!

Link: git.to/codedjlive
(Only works on Firefox, since it uses Audio Data API.)
The app is not most intuitive and has a tiny learning curve.
Please refer to more info section.

Video: http://www.youtube.com/watch?v=bl11T112Jn0

 

A project I did for the PennApps 48-hour hackathon.

About
CodeDJ Live! is a web-based app for programmers to code up visuals in real-time as the music plays. With this app the programmer becomes a Visualizer DJ who produces stunning visuals at a party.

The app is inspired by Iñigo Quilez and his live code demos at SIGGRAPH 2012 Real-time Live event. Similar examples can be seen here and here.

Technical Overview
This app is essentially a WebGL GLSL live coder hooked up to audio spectrum analyzer. The user has access to the GLSL code that is producing the visuals, which can be compiled instantly and displayed.

Technical Details

Music Analyzer
Audio frequency analysis is done using Mozilla's Audio Data API and fast-Fourier transform functions, provided by [blank]'s signal processing library. The Audio Data API provides access to the framebuffer, which contains the decoded audio sample data. This data is fed into the fast-Fourier transform to extract  spectrum frequency data. Frequency data has 1024 channels. This is way too many for the use of this app and needs to be reduced to a manageable and meaningful number. The higher half of the frequencies are barely noticeable, so they are ignored. The lower half are kept and averaged down to 8 channels total for the user to use.

Music Access
Audio Data API requires access to the audio framebuffer data, which is sensitive secured data access. A limitation to this is the "same origin policy", which means it cannot access audio data outside the same root website. This becomes a problem since users need to be able to use their own music for this app. A solution is to host the web app on a public directory on my personal Dropbox, and have users provide their music using their own Dropbox. This is done through the Dropbox javascript API. This works because now the web app and any user provided files are all within the same root website, which is dropbox.com.

Music Streaming
The user logs in through Dropbox API and specifies which folder to contains the music files. Only .ogg and .wav files are accepted, as determined by Mozilla Firefox. Once the user has selected a folder, no music files are downloaded or transferred. Instead, public URLs of the files are generated and stored within the app for that session. These URLs expire within 4 hours. The audio is then streamed per file as they are needed, using the URL. The app does not load more than one audio file at any time.

Visuals Shader
Visuals are generated using a fullscreen quad and a fragment shader. Audio frequency levels are passed into this shader as uniforms. The GLSL code generating the visuals is exposed through a large text editor, where the code can be edited and compiled in real-time to be displayed on the screen.

Postprocessing Shader
Finally, the visuals go through a post-processing shader which applies a series of custom effects to produce dance-club-like styles. This includes over-saturation, bloom, box-blur, and film grain.

Final Deliverables

Here are the final deliverables for the CIS 565 final project.

Live code:
http://iamnop.com/raymarch/

Code
https://github.com/nopjia/WebGL-RayMarch

Slides
http://www.seas.upenn.edu/~pjia/raymarch/presentation.pdf

Report
http://www.seas.upenn.edu/~pjia/raymarch/report.pdf

Video (turn on captions)
http://www.youtube.com/watch?v=djTVUUJvC9E

Some Nice Images

Just throwing up some nice rendered screenshots. 

Click to see full-size (500x400).

Another Experimental Feature

Decided to throw in another fun experimental feature.

The Displaced Bumps option displaces the distance function according to an arbitrary function (in this case a simple sin wave). It doesn't work very well with square shapes. However, it works great with round box, and it creates a very interesting shape with the knot.

Coding Done! (for now)

Worked straight 10 hours today. I think I have reached a point where I am satisfied with the code.

A huge improvement is a user interface for selecting options. To bring up the GUI, press "Shift+Q". The code is live at www.seas.upenn.edu/~pjia/raymarch (since unfortunately iamnop.com is still having issues.) If you are running this, remember to resize your browser to a smaller size for reasonable performance.

Here's what it looks like now:

Aside from fixing corner cases and doing optimizations, here are the new features implemented:

• New GUI! (of course!)
• Mirrored reflections.
• New more realistic soft shadows method, based on technique by the.savage. 
• New fog. Not really fog anymore. It simply darkens the color relative to distance marched. Takes care of the far distances where numerical errors are large. 
• Focal blur, which is basically dithering based on distance. This is implemented last minute and doesn't produce very good results. 

And of course, the project is not complete, and there are a few known bugs:

• Ambient Occlusion and Subsurface Scattering stops working when warping is turned on. 
• As mentioned, focal blur at this moment is an experimental and unfinished feature. To improve, I will need to take several dithered samples and average between them. However, this will hurt performance.
• The quaternion fractal does not render quite well with all modes, since I'm trying to optimize across all types of scenes, balancing quality and performance. The current smoothing level doesn't work quite well for the fractal, so there will be breaks in the surface. It's also running at a few iterations, so it seems to leave artifacts across the scene. I'm just putting it in there as an example of a very complex shape.