Yes, always use HTTPS, even on local networks if there ever are more than one device on the network.
The way to handle services with/without HTTPS is to put everything behind a reverse proxy that automatically handles certificates. Since you’re using NGINX, you can try setup certbot, or switch to something like Caddy that has certificate handling built in. If you have all your Docker containers on one server, you can look at Traefik.



Probably if you move Caddy to the local VM and then forward the ports 80 and 443 to the VPS through WG. So the VPS receives all requests, but instead of going through Caddy first and then through WG, they go straight through WG and into Caddy on your local VM. Then you can access Caddy both locally and externally through the VPS-WG tunnel.
I have no experience setting this up though, I don’t have a VPS in front of my lab.