Grid Tracking is the interesting part of this project. The hope was to get the camera to track the same grid used in calibration and to display the position of the grid in 3D space. This sounds difficult because of the fact that there is only one camera, but it is possible by comparing the grid position with the ones used in calibration. The CalTech toolbox has the ability to infer the position of the grid built in to it which is very cool. This tool can be seen in the normal calibration by using the “Show Extrinsic” button. Therefore, the challenge is to modify the CalTech toolbox to display the grid in “real time”.

The first step in this process is determining the corners of the grid. This is normally done by the user when using the calibration toolbox, but this can obviously not be done if we want automatic tracking. The solution is to use a fiducial on the grid and use it as a reference to find the grid’s location. The biggest problem with this is that the grid cannot be significantly altered, so a small fiducial must be used which are difficult to find in rooms with lots of color. The edges of the grid, specifically the edges corners, are not used in the calibration so they can be used for the fiducial. The image below is the chosen fiducial.

Once the fiducial is found in the code, the centroids are then given to a modified version of CalTech’s code as the corners of the grid. The centroids don’t exactly work out to be the corners, but it is close enough to find the rest of the grid. After that, the rest of the grid is found, like when using the CalTech toolbox normally, and a mesh of points is created. Then this mesh of points is fed into a modified version of the “Show Extrinsic” code mentioned earlier that displays the grid in 3D space. This code is provided below along with sample images.

The output of this code should be the images shown below: