Could this be herpes?

This is extremely difficult for me to ask, so please bare with me. Last august I had sex with my boyfriend at the time in the pool/hot tub/shower and so on and so forth. It was really rough and lasted a long time, and I felt very sore afterwords, partly because he's well endowed. I often feel sore and everything is a little swollen and red down there after sex, but I never thought too much of it, because it would always go back to normal after a day or so. After a couple days the pain didn't fade, and I developed a tiny but extremely painful sore right below my vagina opening. Again, I didn't think too much of it, because thee pool we had had sex in had a lot of chlorine in it. I ended up going to the doctor, and she told me it was herpes. I freaked out because I had been completely faithful to my boyfriend, and had been tested before we started going out. He got tested and found out he didn't have it, and accused ME of cheating on him. For months I've just tried to accept it, but I never had another "outbreak". Ive continued to have sexual relations with people (telling the other party about my problem, of course) and I continue to feel sore afterwards. Heres my issue: because of a medication I take for anxiety, I have a hard time getting aroused, which leads to painful sex. My boyfriend was tested and did not have herpes. I was tested before we went out, and I didn't have sex with anyone else. When my doctor told me I had herpes, she didn't give me a blood test. People have told me that that is the only way to know for sure, but then others have said that the only way I'd know is if I had another outbreak. It's been months, and the only thing keeping me from going to get a blood test is my incredible fear of needles and of being told I have herpes..again. What I'm trying to ask is there any possibility that I don't have it? That all these factors could just be a coincidence, and all I was experiencing was a vaginal tear made worse by a extremely chlorinated pool? Or am I overthinking everything, and should just accept that it's herpes.