- 16 Posts
- 37 Comments
6·3 months agoIt think it’s still unwise to talk about these topics in broad terms like AGI and even “intelligence”. We still have to pick the capabilities apart to have useful discussions about them. I agree these games are better tests than many benchmarks, but it’s also important to note that these solutions use a combination of well-designed deterministic harnesses, as well as LLMs. So it’s inaccurate to say that “LLMs have achieved AGI” (not sure if that’s what you were getting at). This feels like an important milestone, but we’ll have to continue to probe for failure cases in other categories of problems.
Aside from emotional intelligence, experience, embodiment, etc., these ARC-AGI-3 solutions all rely on the sandbox being a safe environment to fail. They iterate through the problem thousands of times before coming to a final solution. Many real-world human problems cannot be re-tried safely or efficiently.
Good reminder to donate to web.archive.org
7·9 months agoI think it’s best to leave the post link as is. I don’t want to point people to misinformation. I’ve added a post description instead with context and the archive link. If people downvote this post because of the 404, so be it.
Ugh, that is utterly disappointing to see from Ars Technica. Here’s a bit of context about it: https://mastodon.social/@nikclayton/116065459933532659
Fortunately, the article was already archived, for what it’s worth: https://web.archive.org/web/20260213194851/https://arstechnica.com/ai/2026/02/after-a-routine-code-rejection-an-ai-agent-published-a-hit-piece-on-someone-by-name/
3.78 Trillion dollar market cap. You don’t get to that valuation by being generous to your customers. Apple is as capitalist as any of them. They’re just wrapped in shiny marketing. Now only if their fans understood that.
The steam/itch situation is a bit complicated. I think the organization behind that action has bad motivations, so it’s not the best example for the long term. My guess is that itch will update their NSFW policy to allow for some NSFW content, but not content that depicts truly heinous material.
The reason I would support that type of activism, to an extent, is that it was effective against Musk when he started turning twitter into a hellscape: https://www.freepress.net/sites/default/files/2022-11/stop_toxic_twitter_coalition_open_letter_to_twitter_final.pdf
I pointed to an actual thing that already happened. How is that fear mongering?
What are your thoughts on this? Is this not an effect on open-source due to regulation? https://www.home-assistant.io/blog/2023/10/13/removal-of-mazda-connected-services-integration/
When I mentioned the TAKE IT DOWN act, we were talking about regulations in general. I did not equivocate laws to EOs. That said, this EO will have effects on regulatory agencies. Therefore, it will have an effect on software as we discussed, open source or not.
When you say that Trump’s EO’s only affects the executive branch, it sounds like you’re trying to say the EO doesn’t matter. My point is that it does matter.
I know this. Are you saying that it doesn’t matter that dozens of government agencies will now have reduced AI regulation? Have you read how many agencies and departments will be affected? https://www.whitehouse.gov/wp-content/uploads/2025/07/Americas-AI-Action-Plan.pdf
I have never once said in this conversation that EOs are law. You insist on bringing that up. What I am saying is that Trump’s EOs do have an actual effect, regardless of what their historical purpose was, and that’s what matters in the real world.
Sure. My understanding of your original comment was that Trump’s EO won’t matter to open source AI because regulations, or lack of regulations, won’t affect open source AI. My point is that regulations do affect open source AI in some significant way.
Is this not a real and direct effect of EOs? Doesn’t it matter that NASA was forced to change their behavior because of EOs? https://www.scientificamerican.com/article/trumps-dei-purge-is-hitting-nasa-hard/
I agree with this, mostly. The tech is real – The blockchain has solid math behind it. I don’t know if I agree with its value though. The viable usecases are significantly limited compared to the hype behind them. Bitcoin for example is still inherently inefficient compared to conventional payment systems. AI feels a lot like that – lots of hype with very little substance at its core.
I’m not sure if this has happened yet, but in theory the TAKE IT DOWN act could be used to shutdown an open source deepfake code or model repository. In that case you’re right that there will be copies that spring up, but I think it is significant that popular projects could be taken down like that.
You’re saying that EO’s are not policy?
ARC Prize maintains multiple tracks around their benchmarks. They have “verified” leaderboards, “community” leaderboards, and they also run the ARC Prize competition.
They update the “verified” leaderboards when they test raw LLMs without sophisticated harnesses. They seem to update this sporadically and only occasionally do press releases or blog posts about new scores. For example, the latest score from Claude Opus 5 (High) is 30% at $20,000, and they didn’t post about that as far as I know. Again, this just the raw LLM without an agentic or world-model harness.
The ARC Prize competition has a harder set of criteria. Participants have to use smaller, open models with a limited compute budget, with open source code, and of course the solutions are verified by ARC Prize at the end of the competition.
The “community” leaderboards, which is what this post is about, are self-reported and not verified by ARC Prize. There are no restrictions on what model is used or limitations on compute. So naturally they aren’t going to make official news releases about those, unless they decide to verify them at some point.
The only reason I chose to post this is that the top solutions seem legitimate, with source code released, and two of them have associated papers.