{"id":15,"date":"2025-01-17T16:45:57","date_gmt":"2025-01-17T16:45:57","guid":{"rendered":"https:\/\/blog.chataignon.org\/joseph\/?p=15"},"modified":"2025-02-19T17:28:18","modified_gmt":"2025-02-19T17:28:18","slug":"why-chatgpt-cant-spell-properly","status":"publish","type":"post","link":"https:\/\/blog.chataignon.org\/joseph\/post-15\/why-chatgpt-cant-spell-properly\/","title":{"rendered":"Why ChatGPT can’t spell"},"content":{"rendered":"\n
disclaimer<\/summary>\n

The experiments presented here were not all carried out at the same time, and the results of a particular test may differ according to ChatGPT updates. They were conducted in French and English with similar results. I used GPT-4o, which did slightly better than Mistral Large and slightly worse than claude-3-haiku.
The tests can be repeated with a Caesar cipher encoder<\/a> and the OpenAI tokenizer<\/a>.<\/p>\n<\/details>\n\n\n\n

Caesar cipher<\/h2>\n\n\n\n

When Caesar, on campaign in Gaul, wrote to the administrator of his domains in Rome, he used a trick to ensure that his messages could only be read by the intended recipient. Each letter was replaced by the letter three places ahead in the alphabet: A became a D, B became an E, and so on. For example, \u00ab\u00a0alea jacta est\u00a0\u00bb becomes \u00ab\u00a0dohd mdfwd hvw\u00a0\u00bb. This simple encryption method is still known as Caesar’s cipher<\/strong>.<\/p>\n\n\n\n

Caesar’s cipher, too simple and too well-known, has long since ceased to offer any security. Nevertheless, I thought it would be an interesting exercise to test ChatGPT’s ability to recognize ciphertext. What would happen if I tried to send an encrypted message, without any context or explanation: would ChatGPT be able to identify the encryption method and to decipher it?<\/p>\n\n\n\n

And that’s where something strange happens. For a human, the difficult part of this task is to identify the code, but once it’s found, decoding the message is straightforward, simply swapping the letters of the message one by one. ChatGPT, on the other hand, while it has no problem finding the code, has a lot of trouble decoding it correctly. In the tests I carried out, ChatGPT generally recognized Caesar’s cipher (with more difficulty in French than in English), was able to explain its principle, and even gave the correspondence between plaintext and cipher letters for the whole alphabet; but decoding is approximate, if not downright false. This is particularly true of long, little-used, lexically isolated words like \u00ab\u00a0gobbledygook\u00a0\u00bb or \u00ab\u00a0rigamarole\u00a0\u00bb.<\/p>\n\n\n\n

So, where does this come from? To understand it, we need to look at the first stage in the processing of text by a Transformer model: tokenization, i.e. the breakdown of text into tokens and the embedding of these tokens.<\/p>\n\n\n\n

Tokenization<\/h2>\n\n\n\n

To perform calculations on text, and thus build language models, it is first necessary to transform the text into numbers. The simplest way to do this is to use the way text is usually encoded on computers, i.e. with a binary code (convertible into a number) associated with each letter. This works, but it’s not very efficient, because to extract meaning from the text you first have to combine the letters into words. The other simple approach is to assign a number per word in the dictionary, rather than per letter. This saves the machine a lot of calculations, but causes other problems:<\/p>\n\n\n\n