After downloading the mod, use a program such as 7zip or WinRAR to extract the main folder of the mod.
Navigate to mods\CustomOSTTracks. The mods folder is supposed to be in your game's installation folder (don't know where that is? click here).
Move the main folder of the mod from step 1 to the folder you navigated to in step 2.
Download and configure CustomOST and extract the ZIP file to your designated tracks folder.