• 1 token = 4 chars in English
knowledge-baseBy Team M37Labs
• 1 token = 4 chars in English
• 1 token= ¾ words
• 100 tokens= 75 words
Or
• 1-2 sentence= 30 tokens
• 1 paragraph= 100 tokens
• 1,500 words= 2048 tokens
To get additional context on how tokens stack up, consider this:
• Wayne Gretzky’s quote “You miss 100% of the shots you don’t take” contains 11 tokens.
• OpenAI’s charter contains 476 tokens.
• The transcript of the US Declaration of Independence contains 1,695 tokens.
https://platform.openai.com/tokenizer
• 1 token
• 100 tokens
Or
• 1-2 sentence
• 1 paragraph
• 1,500 words
To get additional context on how tokens stack up, consider this:
• Wayne Gretzky’s quote “You miss 100% of the shots you don’t take” contains 11 tokens.
• OpenAI’s charter contains 476 tokens.
• The transcript of the US Declaration of Independence contains 1,695 tokens.
https://platform.openai.com/tokenizer