Ah...That sounds a lot like my dating life when I was 20. I, too, had poor self esteem and little self-confidence. On top of that, I'm shy. In my case, it was a result of how I was treated during my growing up years. My narcissistic alcoholic mother was verbally, emotionally and sometimes physically abusive. My father was the perfect enabler who never came to our defense when we were being abused by our mother. He, too, engaged in his share of verbal abuse. If I had to guess, your growing up years probably weren't very happy ones.
I'm also guessing that you so long for attention from men that when you get any that is even just a little positive, you find it hard to pull away even when you know you should. The thing I didn't realize at the time is that others can pick up on your low self esteem and that is attractive to men who are looking for a women who will put up with poor treatment.
Start by telling yourself that you will pull away when a man comes into your life who treats you poorly. It's hard to do, I know, when you feel you get little male attention. When you feel yourself being drawn to such a man and you find yourself thinking that if you turn him away, no one else will come around, stay strong. If it's a situation where you don't pick up on it right away but find out as time goes by, stay strong and pull away when things turn sour.
I'll give you an example --- when I was 19, I met a guy who was a little younger. He was good looking and a good dancer. When he asked me out, I was so thrilled. Within 3 weeks, he was pressuring me to sleep with him. I told him I wasn't ready for that so he backed off. However, soon he was relentless, yet I held my ground. He became manipulative, telling me that I was a prude. Next he told me that if he only wanted sex, he could go to the "combat zone" and pick up a hooker. Finally, he gave me an ultimatum --- either go away for a weekend with him or it was over. I told him I wasn't going to sleep with him just to keep from losing him. He said, he'd ask someone else. I said "fine". Well...an hour later he called me back to give me a "second chance". My answer was still "no". He ranted and raved at what a prude I was, etc.
Looking back on that, I'm amazed and proud of myself for saying "no". Also, I knew it would have only been a matter of time before he dumped me had I slept with him. After I said "no" to him, it became easier to say it to other guys. I met more than my share of them when I was in college. They would ask me out and at the end of the date, ask me to sleep with them. When I said "no", they said "see you around".
Vow to yourself that you will stay strong. Tell yourself that not all attention from males is good attention. When you get treated poorly, break away from him. Lastly, don't give up on finding the right one. Eventually, I met my husband who treated me better than any other man I dated. We've been married 31 years now. You see, I really believe that if you want to be married, you will find the right one. Be patient and have faith.