Use a different prop hanger? I find it a challenge getting that hanger properly anchored to a motor stick. I'd suggest using a wire prop hanger (see: http://www.indoorduration.com/INAVHobbyShopper3.htm
though you'll probably need a slightly larger wire size) or aluminum one (see: http://turnertoys.com/Prop-Hanger-Harlan-021.html
I can make suggestions on how I'd go about it using the Ikara bearing, no guarantee of success. Goal of effective anchor with minimum weight gain.
The problem I have with that anchor is its too small, as you've already found.
If you try to make your motor stick that small you will have problems with it bending unless you use something much stronger than Balsa.
Or if you only shave it down locally, it will be likely to break off and hard to align accurately.
And its PLASTIC, normal glues like Balsa cement or superglue won't hold it well.
So, I'd start with a small piece of harder wood, like spruce or bass. Just wide enough to fit in the notch snug and maybe 3/8 to 1/2 long by 1/8 to 3/16 tall. Grain running along the long dimension. I might not cut this piece to length until the hanger is securely glued on.
Now I'd try one of two glues, make sure they are designed to glue to plastic and wood. Epoxy, medium setting time or polyurethane (Gorilla) glues. I'd try one of each and pick the one with the most secure connection and minimum weight.
Once I had a secure glue joint on the piece of harder wood, I'd trim it to approximately the dimensions above, probably on the smaller side. This gives you a larger surface area of wood that you can use to secure the assembly onto the motor stick. Do that however suits you. You could cut a slot into the stick and glue the assembly into it, or you could just glue it to the side or bottom of the motor stick. If you do the side or bottom, you might want to reinforce the joint by wrapping it in tissue glued on tight to prevent the balsa from splitting.