• 0 Posts
  • 60 Comments
Joined 3 years ago
cake
Cake day: June 9th, 2023

help-circle







  • A token is the word for the base unit of text that an LLM works with. It’s always been that way. The LLM does not directly work with characters; they are collected together into chunks less than a word and this stream of tokens is what the LLM is processing. This is also why the LLMs have such trouble with spelling questions like “how many Rs in raspberry?” — they do not see the individual letters in the first place so they do not know.

    No, the LLMs do not all tokenize the same way. Different tokenizers are (or at least were once) one of the major ways they differed from each other. A simple tokenizer might split words up into one token per syllable but I think they’ve gotten much more complicated than that, now.

    My understanding is very basic and out-of-date.






  • Speaking here as an art noob who generally knows nothing of what the pieces are supposed to mean or what their societal context was when they were made and what forces they were pushing against etc:

    When my arty partner drags me into art museums with huge abstract modern art pieces with just big splotches of heavily textured color (I’m thinking in particular of one giant piece filling a wall with jagged black heaps of paint) they do in fact make me feel feelings.

    In my case, as in OP’s case, they were really bad feelings. I would prefer not to feel really bad and I don’t like that art. But I certainly couldn’t call it ineffective fart-huffing!



  • You need to tell him that. Seriously. Pick some especially bad example to point at, and come to him with, “Hey Bob, I used to get real value from your review comments. I could tell you were thinking about what to say and it helped me to produce better code. Now it seems I am mostly seeing LLM-generated junk like this one that doesn’t help anybody. This isn’t an improvement, can you go back to the more helpful way you used to do things?”