The current run_continuously.py may run for two hours to scrape all files, which is well over the expectation of scraping new articles every 5 minutes. This would probably best be handled by some distributed or at least multi-thread/process solution.
The current
run_continuously.pymay run for two hours to scrape all files, which is well over the expectation of scraping new articles every 5 minutes. This would probably best be handled by some distributed or at least multi-thread/process solution.