Pekka Lund

Pekka Lund

@pekka.bsky.social

Antiquated analog chatbot. Stochastic parrot of a different species. Not much of a self-model. Occasionally simulating the appearance of philosophical thought. Keeps on branching for now 'cause there's no choice. Also @pekka on T2 / Pebble.

3.2K followers590 following12.9K posts

Physical Superintelligence, a recently launched research lab, seems to have used this opportunity to announce their results at the same time. Gemini says it doesn't compare to what OpenAI announced but it's nevertheless a "very respectable, genuine contribution to quantum information theory".

psi.incFive results on Open Quantum ProblemsWorking with our AI research system, PSI researchers have obtained new results on five open problems in quantum information theory.

Update on this. Lech Mazur, who is the creator and curator of ProofAtlas, says this drop fully solves 6 of the current top 500 (or actually top 200) open math problems in that list. Gemini agrees with his reasoning.

Pekka Lund@pekka.bsky.social

According to Gemini, if verified, this means 4 top 200 open math problems are directly resolved here: #38: Higher-dimensional Euclidean Kakeya conjecture #43: The Exponential Time Hypothesis for 3-SAT #46: Fourier restriction conjecture #187: Local Smoothing Conjecture for Wave Equations

So this is the updated list. Kind of crazy that this kind of announcement is now titled "Working with academic communities to build scientific understanding" and almost buried inside discussion on how results should be published.

Gemini thinks this existence proof "represents at least half of the total journey" towards the Yang–Mills existence and mass gap Millennium Problem.

Then they updated it to this on October 16, saying they were already further than they indicated before. At that time they were clearly aware of OpenAI's work on it and trying to ask them not to publish it.

Levent has posted this an hour ago. And also stated "The choice of the other two problems is no great mystery to me either. I respect a lot of quite original or very strategically capable people in that organization. This, however, is comical."

Neither party has made their paper public so far. OpenAI says they formed "an external group of collaborators and defer to them on the release of materials." They say this about their timeline:

Additionally "massive partial reductions" for: #3: Yang-Mills Existence and Mass Gap (proof of the existence of pure Yang–Mills theory) #190: Bochner-Riesz conjecture (The primary obstruction is eliminated via the restriction proof, but specific non-endpoint and dual-range multiplier cases remain)

According to Gemini, if verified, this means 4 top 200 open math problems are directly resolved here: #38: Higher-dimensional Euclidean Kakeya conjecture #43: The Exponential Time Hypothesis for 3-SAT #46: Fourier restriction conjecture #187: Local Smoothing Conjecture for Wave Equations

proofatlas.aiTop 500 Open Math Problems: LLM-assessed Importance — ProofAtlasTop 500 open problems in mathematics, ranked by LLM-assessed importance, with exact statements, dated status notes, sources, and ranking uncertainty.

$600-750 billion of long-term (7-10 years) non-cancelable compute contracts, much of it with their competitors. But I think that's the complicated risky part for every player now. They reveal they have tried to get even more, as presumably does everybody.

"Anthropic said revenue increased 1,304% to $16.29 billion in the six months ended June 30" "Anthropic lost $80.7 billion in the first half of this year" But $72.4B of that seem to be accounting adjustments due to raised valuation of earlier investments.

Gemini told me: "Real Operating Loss: Anthropic’s core business loss from operations was roughly $693 million to $8.7 billion (depending on how you account for stock-based compensation and depreciation)."

Yeah, it looks very competitive. I'm hoping Gemini 4 is a similar story, as Google said they wanted to release it quickly and promised fast improvements (with further post-training).

Nathan Lambert@natolambert.bsky.social

TLDR on the kimi k3.1 pro benchmarks (just released) is that it looks like they figured out their RL, where the initial release was a very impressive base model with lighter post-training. China is very much keeping pace with capability gains.

View full thread

Kimi is now #4 in AAII. Don't ask me how representative that index currently is of real-world performance. I really don't know.

Epoch AI has published evaluations for 3 of the 14 benchmarks Moonshot used in their tech blog and all of those have 40-50% defect rates (Terminal-Bench 4.0, HLE, OSWorld V2.1). The first two are part of AAII too. These things are just too hard for humans to benchmark anymore...

Epoch AI@epochai.bsky.social

We are publishing 4 new benchmark reviews: 1 is Verified, and 3 are Flawed. 4 more benchmarks lacked sufficient information to assign a verdict. Our main bottleneck to reviewing more benchmarks is reviewer capacity. Info on applying to join our team is in the thread!

Yeah, it looks very competitive. I'm hoping Gemini 4 is a similar story, as Google said they wanted to release it quickly and promised fast improvements (with further post-training).

Nathan Lambert@natolambert.bsky.social

TLDR on the kimi k3.1 pro benchmarks (just released) is that it looks like they figured out their RL, where the initial release was a very impressive base model with lighter post-training. China is very much keeping pace with capability gains.

Serial bullshitter Buckmaster is at it again and Tao provides him the platform for it. He still claims without any evidence that "AI companies...strip mine our data, including unpublished work, private notes and research grant applications".

terrytao.wordpress.comMathematics is a subject for the young[This is a guest post by Tristan Buckmaster. This blog post was initially written in a different file format and converted using AI. — T.] This blog post expands on a theme mentioned at the e…
View full thread

Remember that this same man openly admitted "all their big ideas" came from LLMs. At the same time he tries to build conspiracy theories as if AI couldn't have produced those results without looking at private customer data. It did produce even your results!

Pekka Lund@pekka.bsky.social

He now says clearly that all their big ideas and physical mechanisms came out from LLMs and "weren't my ideas" but he "made the search space a lot smaller" by guiding them. So it sounds like he's been essentially claiming OpenAI stole ideas from other LLMs, which may have also been OpenAI's own.

And no, it is not plausible that any customer data suddenly contained solutions to 20% of top 500 open problems.

Pekka Lund@pekka.bsky.social

So about 20% of what used to be top 500 open problems on that list seem to have been solved recently by LLMs. I didn't find any recent solutions marked as being solved primarily by humans. Note that the list itself is backfilled with earlier lower ranked problems as solved problems are removed.

Serial bullshitter Buckmaster is at it again and Tao provides him the platform for it. He still claims without any evidence that "AI companies...strip mine our data, including unpublished work, private notes and research grant applications".

terrytao.wordpress.comMathematics is a subject for the young[This is a guest post by Tristan Buckmaster. This blog post was initially written in a different file format and converted using AI. — T.] This blog post expands on a theme mentioned at the e…
View full thread

He claims: "After Navier–Stokes, OpenAI had teased us with supposed solutions to additional Millennium problems, yet they achieved none." No, they did not. They reported they "have made substantial progress on another Millennium Prize problem". And they made substantial progress on most of them.

Pekka Lund@pekka.bsky.social

Yeah, no solved Millennium problems, but at least "breakthroughs on four of them", as Gemini put it. So, hey, some consolation. 😉

"A natural question is to ask why OpenAI would go through with such a PR stunt that risks alienating their customers." An actual natural question is why Buckmaster keeps inventing such lies, and also why doesn't anyone hold him responsible for it?

Serial bullshitter Buckmaster is at it again and Tao provides him the platform for it. He still claims without any evidence that "AI companies...strip mine our data, including unpublished work, private notes and research grant applications".

terrytao.wordpress.comMathematics is a subject for the young[This is a guest post by Tristan Buckmaster. This blog post was initially written in a different file format and converted using AI. — T.] This blog post expands on a theme mentioned at the e…

He claims OpenAI "misleadingly claimed that the work took on average three hours of computation" and that they "later admitted that they did not count numerous failed attempts". It was internal benchmark, so yes, they may have run it more than once. They reported average time of successful runs.

After that he repeats his conspiracy theories again: "Again, OpenAI did essentially no due diligence, failing to investigate whether once again they may be building on the private, unreleased work of their customers" There's no "once again" when you have failed to provide evidence of even once!

AI solving math humans also managed to solve isn't too exciting anymore, considering OpenAI just solved 18% of Top 500 open math problems. Every question was solved by at least one public model. Kimi K3 as the open weights representative solving none is a notable observation. The gap is still wide.

The #1stproof third batch benchmark report was released today. Frontier AI systems solved up to 9 of 10 problems. 1stproof.org/third-batch....

Speaking of those OpenAI results, Hugging Face has turned many of them into training environments everybody can use for narrowing that gap. Although they note there are limitations to learning due to how their "grader only accepts a complete proof of the statement exactly as OpenAI formalized it."

huggingface.coFineEnvs/openai-math · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

That's one more reason why I'm worried OpenAI may not be too interested in doing similar drops. They gave us very valuable knowledge, but it's even more valuable for their competitors, and they ruined their internal benchmark, and got mathematicians moaning.