Viral infections are often detrimental to host survival and reproduction. Consequently, hosts have evolved a variety of mechanisms to defend themselves against viruses. A component of this arsenal is a set of proteins, termed restriction factors, which exhibit direct antiviral activity. Among these are several classes of proteins (APOBEC3, TRIM5, Tetherin, and SAMHD1) that inhibit the replication of human and simian immunodeficiency viruses. Here, we outline the features, mechanisms, and evolution of these defense mechanisms. We also speculate on how restriction factors arose, how they might interact with the conventional innate and adaptive immune systems, and how an understanding of these intrinsic cellular defenses might be usefully exploited.