The ProPublica/Northpointe racial bias debate, and the broader issue of algorithmic fairness, present significant dilemmas for criminal justice officials, legal practitioners, data scientists, and policymakers. This paper addresses these concerns with a mixed-methods empirical project to inform legal practitioners and data scientists on some of these difficult challenges.