Caddy (reverse proxy) advice

I recently found a guide on using Caddy (reverse proxy) with Plex Media Server (and I guess other stuff I want my viewers to see like Plex Requests).
Plex Requests runs fine, but for some reason when someone streams a video from outside my network, the video will play for like 5 seconds then just suddenly stop.
If I disable Caddy then put traffic straight to the server, it works fine again.
Has anyone else had this experience with Caddy and Plex Server?