It's actually pretty simple. There are 3 things you need to do ...
- In a vertex shader you generally calculate the homogeneous screen coordinates ( gl_Position ) of a vertex by multiplying it with a modelViewProjectionMatrix. For mouse-picking you do the exact opposite. You already know the homogeneous screen coordinates of the mouse, but you want to know the corresponding coordinate ( ray origin ) in your 3D / 2D world.
- A ray also has a direction, getting this direction is the easiest part .. you already calculated it for the view matrix of your camera
- Now that you have the origin and direction of the ray, you basically loop over all objects / meshes you want to check for picking, check whether the ray intersects with it and if-so, calculate the distance between the ray origin and the intersection point. The intersection with the lowest distance is your mouse picking result.