Once the SIFT algorithm has been applied, one has positions of corresponding features in each image. To rebuild each point in 3D space, to undo the projection induceded by the camera, we represent our camera using the pin hole model.This assumption allowas us to draw a line from the camera position to the feature, obtaining an equation for a line that can be used to approximate furthur points down that line beyond our feature. Obtaining equations for each photo and its respective camera we can build a system of equations to see where these two lines intersect. The position of camera one and camera two are varied by some horizontal distance or disparity, similarly as features in photo one and photo two. The figure shows the general idea behind this reconstuction
However this is a “perfect” model that inherently has many flaws. One there is going to be a scale problem because we are not sure if we took the picture with a small or big camera of a small or big “world.” This trick can be seen in old movies when you see Godzilla destroying a city, when it is actually a man in a costume jumping on blocks or when a big building is simply a clay miniature shot up close.
The second problem is that this method requires we know the position and the disparity of the cameras before hand to solve this eqation.
Lastly, due to imperfections that will always exist in a photograph, the corresponding features have both vertical and horizontal disparities, not just one or the other, therefore the lines will not truly ever intersect.