I don't have problems with Plex for being unresponsive. What I did was disable video stream transcoding because the Plex app on my LG tv can play all the formats I send it.
Why you have to disable VPN, I don't know. Not using VPN on the machine my Plex server is on. On an other machine with VPN I can tweak which addresses are send over VPN and which not but don't know if that helps you.
But, if you don't like Plex, maybe you can try something like emby. I don't have experience with that but maybe it is something for you. Something I tried before was serviio. This works but was not as slick as Plex.
Good luck and stay save