Natural, unbleached flannel would of course be the very best thing that you could use - no dyes or chemicals. Flannel is used for castor oil packs for 2 reasons - heat retention, and the nappy, almost fuzzy texture of the fabric grabs and holds oil better than a smooth fabric would, and more of it. Another thing I've noticed is that the flannel seems to cling to my skin better. Not to say that you couldn't use a diaper - sure you could. If you're putting a hot water bottle or a heating pad on it, the heat retention of the fabric is not really all that important - it's going to warm up. And the amount of oil is not really a concern either - if you use a diaper or smoother cotton, and feel that you've absorbed all of the oil, then simply add more.
I get my flannel via an old and much-loved beige flannel sheet that has seen better days. I cut squares off of it.
This thread has inspired me to order more castor oil!