Downloading 20 years of data takes about 4-5 days. This is too slow. Increasing the number of threads past 4 causes increased 429 errors from the dukascopy server so concurrency is capped. Therefore, an alternative ingestion method should be found for when performance is prioritized. However, the current ingestion method works well for low performance machines that do not have the specs to easily process huge chunks of data at once, this will be expanded on in the README.
Downloading 20 years of data takes about 4-5 days. This is too slow. Increasing the number of threads past 4 causes increased 429 errors from the dukascopy server so concurrency is capped. Therefore, an alternative ingestion method should be found for when performance is prioritized. However, the current ingestion method works well for low performance machines that do not have the specs to easily process huge chunks of data at once, this will be expanded on in the README.