Will Refinancing Alone Eliminate PMI?

refinancing

We are looking to refinance from our first mortgage to eliminate PMI and lower interest rate. I was told that since we have not paid the loan down to 80% the value of our home (only had for 2yrs), we wont be able to refinance without continueing to carry it. This is different from what we were told at the signing of our first loan. Is this right?
We currently owe 161,000.00, and financed 165,000.00 @ 6.5%. I did try to refinance which we could do, but the closing costs were outrageous and we would have finished up financing more then we did to start with. Our home is valued between 175k-180k. The lender told us that we could refinance to eliminate PMI in as soon as several months, or pay down to 80%. I had them do again this several times because it was a concern of mine. They definately were not clear and made it sound like we were only paying PMI because we were a first time home buyer.