There's a mysterious new, undocumented model in the https://chat.lmsys.org/ arena chat tool called "gpt2-chatbot" - you can access it by selecting "Direct Chat" and then picking it from the big select menu there
It's providing responses that feel significantly more impressive than GPT-4, for both factual-knowledge lookup and logic puzzles. It's possible this is a stealth preview launch of something like GPT-4.5
It gave me the best ego-prompt response I've seen from any model so far - most models hallucinate something like that I was the CTO of GitHub, for this one every detail it provided was arguably correct (if a little sycophantic towards the end)
@simon@mpesce How do you evaluate these LLMs? I’m not seeing a huge difference, except between small and large models, but even then not as big as I expected. Assuming I’m not using good prompts for this
I have a few prompts I use (starting with the ego-prompt "Who is a Simon Willison?") to get a feel for the size of its knowledge and how likely it is to hallucinate, but for a proper evaluation you really have to spend days of time with it as a regular tool
Or rely on the various benchmarks, but they don't really tell you much about how useful it will be for your own use-cases
@arnicas we know for certain it's not GPT2 because that came out in 2019 and had a fraction of the capability of modern models. I think that name is deliberately a joke. https://en.m.wikipedia.org/wiki/GPT-2
Add comment