The process of aging is associated with broad changes in the brain at the cognitive, neural circuit, and cellular level. Aging of the brain progresses at different rates in the human population, but the genetic basis of those differences has remained unclear. In this study, we searched for a genetic signature of aging by combining genotype data with frontal cortex post-mortem gene expression data from four cohorts: University of Pittsburgh, NIH Braincloud, GTEx, and the Religious Order Study/Memory and Aging project based at Rush University. We found aging-associated genetic variation near synaptic genes, including one SNP that was nominally significant in all cohorts (p <0.01) and genome-wide significant in a meta-analysis (p=8.4E-9). A comparison brain age to Alzheimer’s disease and the underlying genetics showed that brain aging and APOE status are independent predictors of Alzheimer’s disease. Our study provides a systematic framework to uncover the mechanisms that drive brain aging and how it relates to neurodegenerative disorders like Alzheimer’s disease.