OK, this is not my thing at all, but the only .MOV files I know of are the ones that are Apple-format Quicktime videos.
It looks like Instagram does host videos.
Edit: I looked and Instagram is supposed to take .MOV files - maybe the problem is that you are trying to upload video as photos?