Post

another researcher accuses OpenAI of training on conversations and then claiming a breakthrough

Transformer models produced training data text verbatim frequently 2+ years ago. The technology itself has not substantially changed. It's plausible that the lower frequency of verbatim output is the result of mitigation rather than absence. Glad if mathematicians are mad. Coders let this happen.

it’s a model training smell. there’s a series of papers that go into why, but a high quality LLM should only be able to quote *in spirit* not verbatim

I haven't seen it since, but Opus 4.5 on bedrock gave me 2 full, exact paragraphs of Tale of Two Cities, without any I/O AFAIK. It has been extremely hard for me to suspend my skepticism when people tell me LLMs don't plagiarize. Small perturbations to avoid string match actually makes me more mad.

that’s a pretty small excerpt but anyway, the NYT wasn’t able to prove it in court. same as several others not to say it’s impossible, it’s just a bug in the training data if it’s memorizing (arguably not for tale of two cities, since it’s meant to be quotable)

There are a lot of things the judicial and regulatory systems aren't prepared to handle (yet). Anthropic just paid $1.5B. For provenance, yes. It shouldn't be fair use because it competes with the original work. When people tell me it can't quote and it does it's a major reputational problem.

“can’t” is incorrect, it can, but it doesn’t happen often in practice. And when it does it’s on things that are meant to be quoted, like tale of two cities

Earlier I got several full classes from sqlite3. I think that was Microsoft copilot in vscode maybe 2 years ago. Btw I don't push models at all. I'm just trying to do my work.

Personally I think the business model needs to change. Training data needs to be licensed for training use, with real teeth. The vendors shouldn't able to pay a small fine for piracy. Don't even get me started on GitHub. The public acrimony will increase until the vendors reform themselves.

Anthropic paid this fine because they pirated training material.

imo openai should credit him if it was actually a factor, but they’re certainly not required to, he gave up his rights

Sorry if I'm pestering. It shouldn't be easy to accidentally give up his rights. And I think we've seen enough dishonesty from at least OpenAI over years to no longer trust claims they make in public. If I understood how GitHub would be used I wouldn't have participated.