Preventing player from moving out of Camera Viewport

I thought I had this figured out, just use ViewportToWorldPoint to get the world coordinates of the edges of the screen, then compare the player's position to these points and move him if necessary ( you already have world coordinates necessary for moving the player).

However, this only works when the camera is directly overhead at 90 degrees. Changing the angle or FOV messes this up, especially on the sides of the sceens. So anyone know a better approach? If anyone is curious, this is for a top down shooter(SHMUP).