Question about verl vs AReaL

[lzxdjb]

I strictly use the verl-benchmark you provided. And use the code verl-project/verl#2231 to implement a single one-step async.

I use 4 A40 and first 500 case in provided dataset deepscaler_40k_0319.

Below is the verl script I use: both sync mode and async:

python3 -m recipe.one_step_off_policy.async_main_ppo
algorithm.adv_estimator=grpo
data.train_files=/verl/test_data/train.parquet
data.val_files=/verl/test_data/train.parquet
data.train_batch_size=16
data.max_prompt_length=1024
data.max_response_length=2048
data.filter_overlong_prompts=True
data.truncation='error'
data.shuffle=False
actor_rollout_ref.model.path=Qwen/Qwen2.5-3B-Instruct
actor_rollout_ref.actor.optim.lr=3e-6
actor_rollout_ref.hybrid_engine=False
actor_rollout_ref.model.use_remove_padding=False
actor_rollout_ref.actor.ppo_mini_batch_size=2
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1
actor_rollout_ref.actor.use_kl_loss=False
actor_rollout_ref.actor.kl_loss_coef=0.001
actor_rollout_ref.actor.kl_loss_type=low_var_kl
actor_rollout_ref.actor.entropy_coeff=0
actor_rollout_ref.model.enable_gradient_checkpointing=True
actor_rollout_ref.actor.fsdp_config.param_offload=False
actor_rollout_ref.actor.fsdp_config.optimizer_offload=False
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=4
actor_rollout_ref.rollout.tensor_model_parallel_size=1
actor_rollout_ref.rollout.name=vllm
actor_rollout_ref.rollout.gpu_memory_utilization=0.9
actor_rollout_ref.rollout.n=1
actor_rollout_ref.rollout.n_gpus=2
actor_rollout_ref.rollout.load_format=safetensors
actor_rollout_ref.rollout.layered_summon=True
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4
actor_rollout_ref.ref.fsdp_config.param_offload=True
algorithm.use_kl_in_reward=False
trainer.critic_warmup=0
trainer.val_before_train=True
trainer.logger=['console','wandb']
trainer.project_name=async_vs_sync_vs_areal
trainer.experiment_name=2.5B_2048_0.9_async
trainer.n_gpus_per_node=4
trainer.nnodes=1
trainer.save_freq=-1
trainer.test_freq=-1
trainer.total_epochs=1 | tee async.txt

The async time is 900s. The sync time is 1500s

Below is the AReaL script I used:
python3 training/main_async_ppo.py
n_nodes=1 n_gpus_per_node=4
allocation_mode=vllm.d2p1m1+d2p1m1
cluster.fileroot=/ssd11/other/leizh/AReaL/checkpoint
actor.type._class=qwen2
actor.path=Qwen/Qwen2.5-3B
ref.type._class=qwen2.5
ref.path=Qwen/Qwen2.5-3B
dataset.path=/root/.cache/huggingface/hub/datasets--inclusionAI--AReaL-RL-Data/snapshots/07dfc8977909d38366d4e913eb3648a939e5aeb4/data/deepscaler_40k_0319.jsonl
dataset.train_bs_n_seqs=16
group_size=1
actor.sglang.mem_fraction_static=0.9
ppo.gen.max_new_tokens=2048
ppo.ppo_n_minibatches=4
actor_train.mb_spec.max_tokens_per_mb=10240
actor_inf.mb_spec.max_tokens_per_mb=10240
max_concurrent_rollouts=32
max_head_offpolicyness=4
group_adv_norm=True
ppo.adv_norm=False
mem_per_model_worker=22500
mem_per_master_worker=500
mem_per_generation_server=15360
mem_per_gserver_manager=2660
mem_per_rollout_worker=5120
ppo.fuse_rew_ref=False
ppo.kl_ctl=0.0
wandb.project=async_vs_sync_vs_areal
wandb.name=areal
max_head_offpolicyness=8
exp_ctrl.total_train_epochs=1 | tee areal_2.txt

The total time comsumption is 2500s

I am wondering what happens in the AReaL test case. Since I have already set very big for max_concurrent_rollouts and max_head_offpolicyness

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity within the last 14 days.

If this issue is still relevant, please add a comment to keep it open. Otherwise, it will be automatically closed in 16 days.

Thank you for your contribution!