I've been reflecting on LLM tools in IDES, like Sourcegraph's Cody and JetBrains's AI Assistant, and I think that they (and other LLMs targeted for code generation) suffer from the same problem:
The inability to FORGET.
Unless you have complete control over the data used to train the LLMs, you're going to get old solutions for problems that may not exist anymore, because languages, libraries, and frameworks eventually deprecate and remove code, come up with better (or worse, but different) ways to do things, etc.
It's a bit frustrating to get a solution that works, but is the "old" way of doing things (indexed for loop instead of a for-each, or instead of a stream). It's even worse when the solution worked 3 years ago, but no longer does, or worse is a mixture of modern and obsolete code, making it difficult to separate the two (LLMs are entirely based on "mixing", so this happens frequently).
I'm pretty sure the solution for these kinds of tools is to be trained on better (or at least more recent) data.
