Endogenous retroviruses (ERVs) are remnants of ancient retroviral infections of the germ line that can remain capable of replication within the host genome. In the soma, DNA methylation and repressive chromatin keep the majority of this parasitic DNA transcriptionally silent. However, it is unclear how the host organism adapts to recognize and silence novel invading retroviruses that enter the germ line. Krueppel-Associated Box (KRAB)-associated protein 1 (KAP1) is a transcriptional regulatory factor that drives the epigenetic repression of many different loci in mammalian genomes. Here, we use published experimental data to provide evidence that human KAP1 is recruited to endogenous retroviral DNA by KRAB-containing zinc-finger transcription factors (TFs). Many of these zinc-finger genes exist in clusters associated with human chromosome 19. We demonstrate that these clusters are located at hotspots for copy number variation (CNV), generating a large and continuing diversity of zinc-finger TFs with new generations. These zinc-finger genes possess a wide variety of DNA binding affinities, but their role as transcriptional repressors is conserved. We also perform a computational study of the different ERVs that invaded the human genome during primate evolution. We find candidate zinc-finger repressors that arise in the genome for each ERV family that enters the genomes of primates. In particular, we show that those repressors that gained their binding affinity to retrovirus sequences at the same time as their targets invaded the human lineage are preferentially located on chromosome 19 (P-value: 3 × 10−3).