getting the castor oil rinsed out of the flannel?

any suggestions on what to use to get the castor oil rinsed out of the flannel after using it? I've read you can use the same oil soaked pack a few times, just keep it in the fridge inbetween, but after that what to use to wash it? I've tried various soaps and it doesn't get the oil out...what do you all use, or do you just throw it away and start over?