Since the 1980s, the United States has experienced both an increase in income inequality and a rise in residential segregation by income. We develop a general equilibrium model with residential choice, where segregation and inequality amplify each other because of a local spillover affecting educational returns. We calibrate the model to a representative US metro in 1980 and use the micro estimates of neighborhood exposure effects in Chetty and Hendren (2018b) to discipline the spillover. We then explore the economy’s response to an unexpected skill premium shock and show that segregation played a significant role in amplifying the increase in inequality.