Joe provided me with pseudo code and that was much more helpful than reading through other people's code. I actually have something now!
This is optimization with 10 iterations. Apparently mine runs really slow compared to the sample code. I am also checking every pixel for nearest neighbor, which I don't think is what I should be doing. According to the Kwatra paper, I believe they sprinkle the nearest neighbors. I will try that next.
It feels good to have some progress after so much time just reading and trying to figure out someone else's code!