The dysregulation of chromatin and epigenetics has been defined as the overarching cancer hallmark. By disrupting transcriptional regulation in normal cells and mediating tumor progression by promoting cancer cell plasticity, this process has the ability to mediate all defined hallmarks of cancer. In this review, we collect and assess evidence on the contribution of chromatin and epigenetic dysregulation in prostate cancer. We highlight important mechanisms leading to prostate carcinogenesis, the emergence of castration-resistance upon treatment with androgen deprivation therapy, and resistance to antiandrogens. We examine in particular the contribution of chromatin structure and epigenetics to cell lineage commitment, which is dysregulated during tumorigenesis, and cell plasticity, which is altered during tumor progression.