Skip to content
This repository was archived by the owner on Mar 31, 2022. It is now read-only.
This repository was archived by the owner on Mar 31, 2022. It is now read-only.

Format for 'example.txt' Able to read mutliple text lines? #1

Description

@GenTxt

Thanks for the cool repo. Have everything working with custom models and would like to read multiple lines from 'example.txt' and generate max tokens for each pair of title/content lines and/or for each line and save to 'sample.txt'

Currently only reading lines 1 and 2 then generating max tokens and saving to 'sample.txt'

python3 ArticleGenerator.py -f example.txt -o sample.txt -n 1

Generic format of 'example.txt' :
This is line 1 with title text.
This is line 2 with content text.
This is line 3 with new title text.
This is line 4 with new content text.
etc.
Have tried <|startoftext|> , blank lines etc. but only reads lines 1 and 2 then returns them followed by the max tokens.
Is there an example or special format to use?

Cheers

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions