About training data preparation

[Kaiwen-Zhu]

Thanks for the awesome work! However, I have some trouble with data preparation. When I run

sbatch --time=2-00:00:00 --array=0-100%25 --cpus-per-gpu=12 --mem-per-gpu=100G --nodes=1 --gpus-per-node=1 --partition=preempt --job-name=cambrian_precompute_tokens scripts/precompute_tokens_slurm.sh

in unidisc/datasets/preprocessing/webdataset.md, an error occurs: accelerate <command> [<args>] launch: error: the following arguments are required: training_script, which seems to indicate that the script to run (maybe models/datasets/precompute_text_tokens.py ?) is not provided in scripts/precompute_tokens_slurm.sh. Could there be any issue here? Or do I misunderstand it?

Thank you for taking the time to answer！

[alexanderswerdlow]

Hi! Apologies for the confusion:

You should first get the command to run locally, e.g., from the doc:

accelerate launch models/datasets/precompute_tokens.py +experiments='[webdataset,tokenize,vq16_t2i]' data.token_output_dir="/path/to/token_output_dir" data.resolution=512 data.use_chameleon=false loader.batch_size=16 data.raw_data_dir='/path/to/cambrian/jsons/Cambrian10M.jsonl' +model.text_vocab_size=32001 data.img_token_shift=32001 +data.use_identity_collate=true loader.num_workers=2 data.split_dataset=true +data.save_tmp_interval=3600 +data.use_slow_tokenizer=true +data.add_image_token=true

And then, if you want to run w/Slurm, replace accelerate launch with the sbatch prefix:

sbatch --time=2-00:00:00 --array=0-100%25 --cpus-per-gpu=12 --mem-per-gpu=100G --nodes=1 --gpus-per-node=1 --partition=preempt --job-name=cambrian_precompute_tokens scripts/precompute_tokens_slurm.sh

A complete example should be something like:

sbatch --time=2-00:00:00 --array=0-100%25 --cpus-per-gpu=12 --mem-per-gpu=100G --nodes=1 --gpus-per-node=1 --partition=preempt --job-name=cambrian_precompute_tokens scripts/precompute_tokens_slurm.sh models/datasets/precompute_tokens.py +experiments='[webdataset,tokenize,vq16_t2i]' data.token_output_dir="/path/to/token_output_dir" data.resolution=512 data.use_chameleon=false loader.batch_size=16 data.raw_data_dir='/path/to/cambrian/jsons/Cambrian10M.jsonl' +model.text_vocab_size=32001 data.img_token_shift=32001 +data.use_identity_collate=true loader.num_workers=2 data.split_dataset=true +data.save_tmp_interval=3600 +data.use_slow_tokenizer=true +data.add_image_token=true

If you look at the slurm script, you will see the $@ which just takes any number of script args (see here).

Hope this helps!

[Kaiwen-Zhu]

Thanks for your swift response! However, I still encounter an error in precompute_tokens.py: hydra.errors.MissingConfigException: In 'config': Could not load 'experiments/tokenize'. It seems that there should be tokenize.yaml in configs/experiments.

Additionally, may I ask that if I want to use WebDataset without precomputing tokens (i.e., I only have tar files containing jpg, json, txt as in the provided synthetic dataset), what should be the configuration like? The massive config options really confuse me a lot. Would it be possible to provide a toy example or highlight the key settings required for this use case? Thanks for your consideration!