Drill press table fence with cam levers