After downloading the mod, use a program such as 7zip or WinRAR to extract the main folder of the mod.
Navigate to mods\CustomOSTTracks. The mods folder is supposed to be in your game's installation folder (don't know where that is? click here).
Move the main folder of the mod from step 1 to the folder you navigated to in step 2.
It may be advised to change some CustomOST settings to prevent game crashes, namely preventing songs from being preloaded. You can also try applying the 4GB Patch to your exe file in order to allow it more RAM before overloading and crashing.