Right on the money. One of the big things with AI safety is "we have no fucking clue how AGI can originate so we are constantly in the dark." If we ever did create it, we likely would not immediately know it was AGI, and that creation could go very terribly in a number of ways.

Whenever I hear someone say that something is impossible with current technology, I think about my grandma. When she was a kid, only some important people had telephones. Doctors, police, etc.

In her lifetime we went from that to today, and, since she's still alive, even further into the future.

Whenever someone calls something impossible, I think about how far technology will progress in my own lifetime and I know that they've got no idea what they're talking about. (Unless, like you said, it's against the laws of physics. But sometimes even then I'm not so sure, cause it's not like we understand those entirely. )

Actually, we do already know that we're close to a theoretical limit of increasing computing power as we currently know it. The transistor can't really get that much smaller, before it stops working.

Also, if you're talking about the article as linked, that is a mere introduction to a much longer paper.

The fact that human brain is capable of general intelligence tells us everything we need to know about the processing power needed to run one.

The article did mention a fundamental obstacle. It said quite clearly that we would run out of resources before we had enough computing power. I suppose you could counter that by arguing that we could discover magic, or magical technology, or a lot of new resources through space exploration.

Of course things get more efficient. But in the past few decades they've gotten efficient in predictable, and mostly predicted, ways. It's certainly possible that totally unexpected things can happen. I could win the lottery next week. Is that the standard? Are you pushing the stance that says AGI is somewhat less likely than winning the lottery or getting struck by lightning, but by golly it's more than zero, how dare you suggest that it's anywhere close to zero?

The paper's scope is to prove that AI cannot feasibly be trained, using training data and learning algorithms, into something that approximates human cognition.

The limits of that finding are important here: it's not that creating an AGI is impossible, it's just that however it will be made, it will need to be made some other way, not by training alone.

Our squishy brains (or perhaps more accurately, our nervous systems contained within a biochemical organism influenced by a microbiome) arose out of evolutionary selection algorithms, so general intelligence is clearly possible.

So it may still be the case that AGI via computation alone is possible, and that creating such an AGI will not require solution of an NP-hard problem. But this paper closes one potential pathway that many believe is a viable pathway (if the paper's proof is actually correct, I definitely am not the person to make that evaluation). That doesn't mean they've proven there's no pathway at all.

A breakthrough in quantum computing wouldn't necessarily help. QC isn't faster than classical computing in the general case, it just happens to be for a few specific algorithms (e.g. factoring numbers). It's not impossible that a QC breakthrough might speed up training AI models (although to my knowledge we don't have any reason to believe that it would) and maybe that's what you're referring to, but there's a widespread misconception that Quantum computers are essentially non-deterministic turing machines that "evaluate all possible states at the same time" which isn't the case.

@ContrarianTrail @JRepin well I guess somebody would first need to clearly define what "AGI" is. Currently it's just "whatever the techbro hypers want it to be".

And then there's the matter (ha!) of your assumption that we understand all laws of physics necessary that "matter obeys", or that we can reasonably understand them. That's a pretty strong assumption: individual human minds are pretty limited and communication adds overhead, and we might reach a point where we're stuck.

incremental improvements on a dead end, still gets you to the dead end.

Another possibility is that humans just aren't smart enough to figure out AGI. While I'm sure that we will continue incrementally improving technology in some form, it's not at all self-evident that these improvements will eventually add up to AGI.

If there were a giant asteroid hurling toward Earth, set to impact sometime in the next 20 to 200 years, I’d say there’s definitely a need for urgency. A true AGI is somewhat of an asteroidal impact in itself.

Not really a good comparison. The steam engine was an extant technology at that point. AGI is not, and we really no idea if/when it will be. One thing is clear though, it is not as close on the horizon as tech bros want us to think it is.

Like wrecking the biosphere in its persuit.