That sound could have been due to a motor height issue if everything else seems okay inside. Check the bevel and spur gear teeth to make sure you haven't stripped them as well. When you put it back together there's a small hex screw under the grip. This controls the motor height which is important for the spur and bevel gear to mesh properly.
In terms of shimming... no one can really give you a definitive answer on what's right. Shimming depends on the individual mechbox characteristics. You want the gears to be able to move freely while not moving back and forth too much.
http://www.mechbox.com has a video in their V2 mechbox section briefly explaining it. It's trial and error until you get a good fit really.