It was like ptsd in a way.
It wasn't just 'like' PTSD, it possibly was actual PTSD! I received a PTSD diagnosis several months after Dday. It's very common for people going through betrayal trauma.
I can relate to your story. I kicked out my now XH when he told me about the affair and had NC with him except through email for things related to the house. However, there was one day when I saw him. He was moving his stuff out of the house and so I cleared out and left my cousin there to watch the situation. While I was out, my cousin called to say one of my dogs had been hurt, so I rushed home (don't worry, dog ended up totally fine). In my panic over my dog, I forgot my X would be there. I saw him and oddly felt nothing at the moment, but later experienced all the panic you describe.
Going through this reminded me of experiencing grief after my mom died. It comes in waves; it hits you hard and just when you think you can't take it for one more second...it subsides, at least a little. It does this over and over, until the waves come less and less frequently. Now, 3.5 years out and divorced, it comes rarely. Sometimes I miss the life we had together or how he treated me (at least to my face!), but I could never have had that life again, even if I had stayed. That life was over the second I found out about the A. I find it best to think of it like a death...it's normal to wish the situation was different, but knowing that it is over.
Good luck to you! As others said, you are still very early in this process. It does get better!