I am working with German Texts, where I need to tokenize texts using GPT2Tokenizer.

To tokenize the text, I wrote the implementation as follows:

from transformers import GPT2Tokenizer

text = "zügiger Transport des ABCD stabilen Kindes in die Notaufnahme UKA"
text = text.encode("utf-8").decode("utf-8")  # Re-encode to fix encoding issues

# Load GPT-2 tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Tokenize the text
tokens = tokenizer.tokenize(text)

print(tokens)  # Should properly tokenize "zügiger" instead of splitting "ü"

Now, when I execute this code snippet I get output as follows:

['z', 'ü', 'g', 'iger', 'ĠTransport', 'Ġdes', 'ĠABC', 'D', 'Ġstabil', 'en', 'ĠKind', 'es', 'Ġin', 'Ġdie', 'ĠNot', 'au', 'fn', 'ah', 'me', 'ĠUK', 'A']

After a bit of analysis, I have found that all German language specific characters are mis-decoded as Latin-1 see the table below.

| Character | UTF-8 Bytes | Misdecoded as Latin-1 | Resulting String |
|-----------|-------------|-----------------------|------------------|
| ä         | C3 A4       | à + ¤                 | ä               |
| ö         | C3 B6       | à + ¶                 | ö               |
| ü         | C3 BC       | à + ¼                 | ü               |
| ß         | C3 9F       | à + Ÿ                 | ß               |

Now, how I can keep German language specific characters like (ä, ö, ü, ß) inside tokens after the tokenization process, avoiding unintentional misdecodeding, i.e. "zügiger" becomes something like ['z', 'ü', 'g', 'iger'].

I am working with German Texts, where I need to tokenize texts using GPT2Tokenizer.

To tokenize the text, I wrote the implementation as follows:

from transformers import GPT2Tokenizer

text = "zügiger Transport des ABCD stabilen Kindes in die Notaufnahme UKA"
text = text.encode("utf-8").decode("utf-8")  # Re-encode to fix encoding issues

# Load GPT-2 tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Tokenize the text
tokens = tokenizer.tokenize(text)

print(tokens)  # Should properly tokenize "zügiger" instead of splitting "ü"

Now, when I execute this code snippet I get output as follows:

['z', 'ü', 'g', 'iger', 'ĠTransport', 'Ġdes', 'ĠABC', 'D', 'Ġstabil', 'en', 'ĠKind', 'es', 'Ġin', 'Ġdie', 'ĠNot', 'au', 'fn', 'ah', 'me', 'ĠUK', 'A']

After a bit of analysis, I have found that all German language specific characters are mis-decoded as Latin-1 see the table below.

| Character | UTF-8 Bytes | Misdecoded as Latin-1 | Resulting String |
|-----------|-------------|-----------------------|------------------|
| ä         | C3 A4       | à + ¤                 | ä               |
| ö         | C3 B6       | à + ¶                 | ö               |
| ü         | C3 BC       | à + ¼                 | ü               |
| ß         | C3 9F       | à + Ÿ                 | ß               |

Now, how I can keep German language specific characters like (ä, ö, ü, ß) inside tokens after the tokenization process, avoiding unintentional misdecodeding, i.e. "zügiger" becomes something like ['z', 'ü', 'g', 'iger'].

• python
• nlp
• tokenize
• large-language-model
• gpt-2

Share Follow edited Mar 4 at 18:05 Christoph Rackwitz 15.9k55 gold badges3939 silver badges5151 bronze badges asked Mar 3 at 22:32 RajibTheKingRajibTheKing 1,36211 gold badge1515 silver badges3737 bronze badges 1

• 1 Is the text encoded latin-1 to start with? – JonSG Commented Mar 3 at 22:38

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Try direct encoding and decoding with the Tokenizer

from transformers import GPT2Tokenizer

text = "zügiger Transport des ABCD stabilen Kindes in die Notaufnahme UKA"

# Load GPT-2 tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# Tokenize using encode and decode
encoded_tokens = tokenizer.encode(text, add_prefix_space=False)
decoded_tokens = [tokenizer.decode(token_id) for token_id in encoded_tokens]

print(decoded_tokens)

Result

['z', 'ü', 'g', 'iger', ' Transport', ' des', ' ABC', 'D', ' stabil', 'en', ' Kind', 'es', ' in', ' die', ' Not', 'au', 'fn', 'ah', 'me', ' UK', 'A']

Using another text

text = "Die Straßen sind nass"

['Die', ' Stra', 'ß', 'en', ' s', 'ind', ' n', 'ass']