Realized I was feeding my GPT wrong data for months

I was feeding my AI assistant whole webpage transcripts to summarize reports from our Atlanta office. Took me 4 months to figure out the AI was getting confused by all the navigation menus and footer text. What tipped me off was when I compared a summary from a clean text file versus a full webpage dump - the difference was night and day. Has anyone else noticed a big jump in output quality after cleaning up their input data?

2 comments

2 Comments

oliver71913d ago

Webpage scraping is its own headache. Navigation and boilerplate kill context for sure. Clean text vs raw HTML is a totally different ballgame. The AI picks up way more signal from the stripped down version. Four months is rough though. Most people don't realize how much garbage is in those full page dumps until they compare side by side.

terryb1113d ago

Wait is this really that big of a deal? I scrape pages all the time and never bother stripping them down and it still works fine lol.