Finally got my Python script to run 5 times faster

I was trying to pull data from a website for a small project. For a month, I used a method that checked each page one by one, and it took about 20 minutes to finish. Yesterday, I tried using the 'requests' library with a simple loop instead of my old way, and the whole thing ran in under 4 minutes. The difference was just learning to let the library handle the connections. Has anyone else had a big speed boost from switching a basic method?

3 comments

3 Comments

masonm8516d ago

That jump from 20 minutes to 4 is huge. Were you using something like Selenium or BeautifulSoup for the old method? I've seen that happen when people use a full browser driver for simple data pulls. Did you have to change how you parsed the data after switching to requests, or was it a straight swap?

christopher_roberts296d ago

Yeah, it was pure Selenium before, total overkill like patriciasingh said. The parsing was mostly the same once I got the raw HTML with requests. Just felt dumb for not trying it sooner.

patriciasingh16d ago

Oh man, I feel that. My first scraper used Selenium to click a single button over and over. Like using a crane to push a doorbell. Switched to just posting to the form endpoint with requests and it felt like magic. Took it from an hour to maybe ten seconds.