Assess NCBI FTP pipeline for changes necessary to support raw data storage

[ialarmedalien]

Assess the NCBI FTP pipeline for any changes necessary to store downloaded data in the form described in #218.

• if data just needs to be copied from the CTS output dir (aka some location in the cts bucket) (possibly with the addition of checksums), can an s3 batch job be used?
• what other steps are required?

[mattldawson]

Put on hold pending a decision of getting a versioned bucket for ref-data in the lakehouse,which would allow us to just use rclone or a similar tool for periodic syncing NCBI records.

If we do move forward with these pipeline changes, here's what I think would be required at a high level:

• The manifest generation and download steps should remain mostly or entirely unchanged
• The archiving functions can be removed
• The destination key prefix for downloaded files and metadata would need to include the version folder (e.g., refseq_123). It might make sense to have a config setting for just this string, which gets automatically embedded in the full key (e.g., tenant-general-warehouse/refdata/datasets/ncbi/raw_data/refseq_123/<files>