Turns Out I'm Missing Something

The texture optimization iteration in fact has two steps:

1. optimize the texture energy fuction
2. find nearest neighborhood and replace

I only did #2. I'm still reading and Googling how exactly #1 works and how to do it...