As for file type, I have no idea. JPG or other standard types should do fine.
And I'll give the tutorial another go:
When you click the billboards to select images, a meny window will open. The game will look in the RCT3 sub-folder in "my pictures". However, the "My pictures" folder doesn't exist in Vista. Therefore, click the "up one level" button to do just that.
You'll then find yourself in the C:user/documents area. You'll notice many folders that are in your Documents folder, and among these, "my pictures", "My videos", etc. However, none of these are valid choices, they are only there because RCT3 thinks they are. In XP, "My Pictures" and all its cousins are sub-folders in Documents. In Vista, they are separate folders. Therefore, ignore all folders and go up another level.
You'll then be somewhere in the C: drive. Select your Pictures folder from there. Whatever you've put in that folder should appear. Then, just select image and place the billboard. That should be it.