An alternate mode (commonly shortened to "altmode" or "alt mode") is a term used to describe a Transformer's non-humanoid form, whether it be vehicle, animal, or otherwise.
Some Transformers don't have a humanoid form, so in those special cases, the altmode is perceived to be the mode which is more geared towards utility or disguise while their "primary" mode is the one geared more towards interaction with other Transformers. For example, for the G1 characters Laserbeak and Ravage, their animal forms are their primary mode, and the cassettes their alternate modes. Even Grand Slam and Raindance, who transform from cassettes to vehicles, would be thought of as having a cassette alternate mode, as their vehicle forms -- while not humanoid or anamalistic -- are at least self-mobile.
Transformers with more than two modes are usually considered to have multiple altmodes. (Punch/Counterpunch on the other hand has one altmode and two robot modes.)
The earliest recorded use of the term "alt mode" is an alt.toys.transformers post from March 1994 [1]. From the context, it appears that the term was already understood by fans at that time, as the post's author does not explain what he means.