In tables special characters gets replaced with strange characters, the charset is not managed properly

[lavjamanxd]

Issue
In tables if we are trying to use special characters, after formatting it will replace special characters to strange ones.

How to reproduce

1. Try to format the attached test.file
   test-ansi.zip
   

[antham]

Thanks for reporting.

This is linked to your first ticket right ? What seems to be wrong is that tables don't support specials characters because the rest of the file is ok.

[lavjamanxd]

I'm not sure if its connected to the first one. We just converted our files to extended ANSI (1252), after that we were able to run ghokin on them and then we noticed this one.

So, it happens in tables only.
That's why i tried to put that test string into multiple places in the file, to check where it can happen.

[antham]

Ok so could you provide me a file for the first ticket like in this one to reproduce the problem.

About this ticket, I guess the problem come from a library used in ghokin, not ghokin itself, it's going to be a bit tricky to fix if I am correct.

[lavjamanxd]

Done, attached the file for the other issue.
Thanks!

[lavjamanxd]

update:
This happens only if the Encoding is ANSI.
Formatting a file which has UTF-8 without BOM Encoding, works fine!

[antham]

So sorry to address this issue now, I wasn't able to figure out a proper fix at the time and left this aside in a corner of my mind.

My 2 cts about this issue is that we must manage only UTF-8 files and nothing else, it can sounds a bit let say "unfair" (can't find the appropriate word) but managing charset is quite a nightmare. I think this is the more straightforward and error-bug-free solution we can have.

So I added an automatic converter that will detect and change the charset of the file to UTF-8 and added a warning in the README, everything is in this release => https://github.com/antham/ghokin/releases/tag/v3.4.2