The first coaster to give me that sort of feeling was when I was a child, about 5 or 6, and I rode Space Mountain at Disney World. This was the year it opened, and neither me, my mom or her friend we were with had any idea that it was a coaster. Well, when the ride was over I had to just about pry my hands loose from the lap bars and I could have swore that I had barely avoided flying right off into the darkness - and that I had gone upside down I don't know how many times! Unfortunately for me, that ride sparked a couple of decades of coaster phobia (I was most terrified of inverted coasters even though of course Space Mountain at WDW has no inversions).
The next time was roughly 20 years later when someone (a different friend of my mom's) dragged me onto the Mind Eraser at Six Flags America (which was known as Adventure World back then). I could have swore I was going to die....you don't know how bad I was hoping it would break down before we got on so I could save face....But of course, it didn't.
Then, after I had gotten over my phobia and became a coaster fan, I was at Kings Island and rode Firehawk. Now I had ridden the other two Vekoma flyers (Stealth at Great America and Batwing at Six Flags America) and never had an issue. But that night I swore I was going to be flung out and go splat on the ground below. That ride had to be about the longest two minutes of my whole day at the park....I was death gripping the whole time and was quite relieved when the ride came to an end. I have no idea what brought that on.
A couple mentions to both Millennium Force and Top Thrill Dragster. I rode both for the first time upon my first visit to Cedar Point 20 years ago. I momentarily freaked out on Millennium Force upon seeing the lift hill seeming to disappear into blackness on my first ride, which was at night. That was unnerving. I had ridden TTD that morning and had never been on any coaster that fast or tall and needless to say I was more than a bit nervous. When I got off my hands were shaking....
The only other time on a coaster I had any sort of feelings of danger was the first time I rode Skyrush. I was there for an ACE event and I waited until night ERT for my first ride as the park was pretty busy. That first ride was a totally hair raising sensation that I was going to be pulled right out of my seat. But I got right back on and rode some more and it is still my #1 coaster.
I am going to make a brief mention of a non-coaster ride that has freaked me out in the past (and to a degree still does) - the skyride at Knoebels. I don't know what it is about that ride but the first time I rode it, it was worse than Firehawk. I was totally feeling like "OMG I'm going to fall out!" I am not afraid of heights but to me there is something totally sketchy appearing about that ride. I've ridden it since then, and I always ride it when I go to Knoebels. But still, there lingers in the back of my mind the irrational feeling or fear that I might fall off...