Ever start with too fine a grit? Honestly, I did the same thing on a table and it left those little marks. You gotta work through the grits real slow with oak. Tbh, I begin with 120, even if it feels rough, and only move up after the scratches are gone. Takes forever but it's the only way to get it smooth.