I hope some of the other suggestions in this thread help. Maybe OP is reducing the amount of something necessary in your bloodstream that you are not replacing.
As you say it is the white stuff you want to get rid of in your mouth, you may want to try swishing for less time but adding a few drops of oregano or tea tree oil to the regular oil to kill off some of the bacteria on your tongue.
Do you clean your tongue? If not a light brush / scrape with tongue cleaner might do the trick.