Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion training/README_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Anima model is trained based on QLoRA's [33B guanaco](https://huggingface.co/tim
We mainly use the Chinese training dataset put together by project [Chinese-Vicuna](https://github.com/Facico/Chinese-Vicuna):  [guanaco_belle_merge_v1.0](https://huggingface.co/datasets/Chinese-Vicuna/guanaco_belle_merge_v1.0) in our finetune training work.

* **Rationale**:
According to the conclusions in [QLoRA] (https://arxiv.org/abs/2305.14314)Appendix B.4 and Table 9's Grid Search: For QLoRA fine-tuning, a larger number of training samples is not necessarily better. 10,000 steps is a size with a relatively good ROI. Therefore, we want to choose a dataset with no less than 10,000 steps. The [Belle 10M](https://github.com/LianjiaTech/BELLE/blob/main/data/10M) dataset seems too big, and the data quality is unclear to us. Due to limited time, we will first choose guanaco_belle_merge_v1.0. Later, we will further test more datasets and the effects of data quality filtering in a more systematic way.
According to the conclusions in [QLoRA](https://arxiv.org/abs/2305.14314) Appendix B.4 and Table 9's Grid Search: For QLoRA fine-tuning, a larger number of training samples is not necessarily better. 10,000 steps is a size with a relatively good ROI. Therefore, we want to choose a dataset with no less than 10,000 steps. The [Belle 10M](https://github.com/LianjiaTech/BELLE/blob/main/data/10M) dataset seems too big, and the data quality is unclear to us. Due to limited time, we will first choose guanaco_belle_merge_v1.0. Later, we will further test more datasets and the effects of data quality filtering in a more systematic way.

* **Acknowledgement**:Thanks [Chinese-Vicuna](https://github.com/Facico/Chinese-Vicuna)、[Belle](https://github.com/LianjiaTech/BELLE)、[GuanacoDataset](https://huggingface.co/datasets/JosephusCheung/GuanacoDataset) for their contributions to all the open datasets。

Expand Down