Artwork

内容由Machine Learning Street Talk (MLST)提供。所有播客内容(包括剧集、图形和播客描述)均由 Machine Learning Street Talk (MLST) 或其播客平台合作伙伴直接上传和提供。如果您认为有人在未经您许可的情况下使用您的受版权保护的作品,您可以按照此处概述的流程进行操作https://zh.player.fm/legal
Player FM -播客应用
使用Player FM应用程序离线!

Jürgen Schmidhuber - Neural and Non-Neural AI, Reasoning, Transformers, and LSTMs

1:39:39
 
分享
 

Manage episode 436675796 series 2803422
内容由Machine Learning Street Talk (MLST)提供。所有播客内容(包括剧集、图形和播客描述)均由 Machine Learning Street Talk (MLST) 或其播客平台合作伙伴直接上传和提供。如果您认为有人在未经您许可的情况下使用您的受版权保护的作品,您可以按照此处概述的流程进行操作https://zh.player.fm/legal

Jürgen Schmidhuber, the father of generative AI shares his groundbreaking work in deep learning and artificial intelligence. In this exclusive interview, he discusses the history of AI, some of his contributions to the field, and his vision for the future of intelligent machines. Schmidhuber offers unique insights into the exponential growth of technology and the potential impact of AI on humanity and the universe.

YT version: https://youtu.be/DP454c1K_vQ

MLST is sponsored by Brave:

The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at http://brave.com/api.

TOC

00:00:00 Intro

00:03:38 Reasoning

00:13:09 Potential AI Breakthroughs Reducing Computation Needs

00:20:39 Memorization vs. Generalization in AI

00:25:19 Approach to the ARC Challenge

00:29:10 Perceptions of Chat GPT and AGI

00:58:45 Abstract Principles of Jurgen's Approach

01:04:17 Analogical Reasoning and Compression

01:05:48 Breakthroughs in 1991: the P, the G, and the T in ChatGPT and Generative AI

01:15:50 Use of LSTM in Language Models by Tech Giants

01:21:08 Neural Network Aspect Ratio Theory

01:26:53 Reinforcement Learning Without Explicit Teachers

Refs:

★ "Annotated History of Modern AI and Deep Learning" (2022 survey by Schmidhuber):

★ Chain Rule For Backward Credit Assignment (Leibniz, 1676)

★ First Neural Net / Linear Regression / Shallow Learning (Gauss & Legendre, circa 1800)

★ First 20th Century Pioneer of Practical AI (Quevedo, 1914)

★ First Recurrent NN (RNN) Architecture (Lenz, Ising, 1920-1925)

★ AI Theory: Fundamental Limitations of Computation and Computation-Based AI (Gödel, 1931-34)

★ Unpublished ideas about evolving RNNs (Turing, 1948)

★ Multilayer Feedforward NN Without Deep Learning (Rosenblatt, 1958)

★ First Published Learning RNNs (Amari and others, ~1972)

★ First Deep Learning (Ivakhnenko & Lapa, 1965)

★ Deep Learning by Stochastic Gradient Descent (Amari, 1967-68)

★ ReLUs (Fukushima, 1969)

★ Backpropagation (Linnainmaa, 1970); precursor (Kelley, 1960)

★ Backpropagation for NNs (Werbos, 1982)

★ First Deep Convolutional NN (Fukushima, 1979); later combined with Backprop (Waibel 1987, Zhang 1988).

★ Metalearning or Learning to Learn (Schmidhuber, 1987)

★ Generative Adversarial Networks / Artificial Curiosity / NN Online Planners (Schmidhuber, Feb 1990; see the G in Generative AI and ChatGPT)

★ NNs Learn to Generate Subgoals and Work on Command (Schmidhuber, April 1990)

★ NNs Learn to Program NNs: Unnormalized Linear Transformer (Schmidhuber, March 1991; see the T in ChatGPT)

★ Deep Learning by Self-Supervised Pre-Training. Distilling NNs (Schmidhuber, April 1991; see the P in ChatGPT)

★ Experiments with Pre-Training; Analysis of Vanishing/Exploding Gradients, Roots of Long Short-Term Memory / Highway Nets / ResNets (Hochreiter, June 1991, further developed 1999-2015 with other students of Schmidhuber)

★ LSTM journal paper (1997, most cited AI paper of the 20th century)

★ xLSTM (Hochreiter, 2024)

★ Reinforcement Learning Prompt Engineer for Abstract Reasoning and Planning (Schmidhuber 2015)

★ Mindstorms in Natural Language-Based Societies of Mind (2023 paper by Schmidhuber's team)

https://arxiv.org/abs/2305.17066

★ Bremermann's physical limit of computation (1982)

EXTERNAL LINKS

CogX 2018 - Professor Juergen Schmidhuber

https://www.youtube.com/watch?v=17shdT9-wuA

Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability (Neural Networks, 1997)

https://sferics.idsia.ch/pub/juergen/loconet.pdf

The paradox at the heart of mathematics: Gödel's Incompleteness Theorem - Marcus du Sautoy

https://www.youtube.com/watch?v=I4pQbo5MQOs

(Refs truncated, full version on YT VD)

  continue reading

185集单集

Artwork
icon分享
 
Manage episode 436675796 series 2803422
内容由Machine Learning Street Talk (MLST)提供。所有播客内容(包括剧集、图形和播客描述)均由 Machine Learning Street Talk (MLST) 或其播客平台合作伙伴直接上传和提供。如果您认为有人在未经您许可的情况下使用您的受版权保护的作品,您可以按照此处概述的流程进行操作https://zh.player.fm/legal

Jürgen Schmidhuber, the father of generative AI shares his groundbreaking work in deep learning and artificial intelligence. In this exclusive interview, he discusses the history of AI, some of his contributions to the field, and his vision for the future of intelligent machines. Schmidhuber offers unique insights into the exponential growth of technology and the potential impact of AI on humanity and the universe.

YT version: https://youtu.be/DP454c1K_vQ

MLST is sponsored by Brave:

The Brave Search API covers over 20 billion webpages, built from scratch without Big Tech biases or the recent extortionate price hikes on search API access. Perfect for AI model training and retrieval augmentated generation. Try it now - get 2,000 free queries monthly at http://brave.com/api.

TOC

00:00:00 Intro

00:03:38 Reasoning

00:13:09 Potential AI Breakthroughs Reducing Computation Needs

00:20:39 Memorization vs. Generalization in AI

00:25:19 Approach to the ARC Challenge

00:29:10 Perceptions of Chat GPT and AGI

00:58:45 Abstract Principles of Jurgen's Approach

01:04:17 Analogical Reasoning and Compression

01:05:48 Breakthroughs in 1991: the P, the G, and the T in ChatGPT and Generative AI

01:15:50 Use of LSTM in Language Models by Tech Giants

01:21:08 Neural Network Aspect Ratio Theory

01:26:53 Reinforcement Learning Without Explicit Teachers

Refs:

★ "Annotated History of Modern AI and Deep Learning" (2022 survey by Schmidhuber):

★ Chain Rule For Backward Credit Assignment (Leibniz, 1676)

★ First Neural Net / Linear Regression / Shallow Learning (Gauss & Legendre, circa 1800)

★ First 20th Century Pioneer of Practical AI (Quevedo, 1914)

★ First Recurrent NN (RNN) Architecture (Lenz, Ising, 1920-1925)

★ AI Theory: Fundamental Limitations of Computation and Computation-Based AI (Gödel, 1931-34)

★ Unpublished ideas about evolving RNNs (Turing, 1948)

★ Multilayer Feedforward NN Without Deep Learning (Rosenblatt, 1958)

★ First Published Learning RNNs (Amari and others, ~1972)

★ First Deep Learning (Ivakhnenko & Lapa, 1965)

★ Deep Learning by Stochastic Gradient Descent (Amari, 1967-68)

★ ReLUs (Fukushima, 1969)

★ Backpropagation (Linnainmaa, 1970); precursor (Kelley, 1960)

★ Backpropagation for NNs (Werbos, 1982)

★ First Deep Convolutional NN (Fukushima, 1979); later combined with Backprop (Waibel 1987, Zhang 1988).

★ Metalearning or Learning to Learn (Schmidhuber, 1987)

★ Generative Adversarial Networks / Artificial Curiosity / NN Online Planners (Schmidhuber, Feb 1990; see the G in Generative AI and ChatGPT)

★ NNs Learn to Generate Subgoals and Work on Command (Schmidhuber, April 1990)

★ NNs Learn to Program NNs: Unnormalized Linear Transformer (Schmidhuber, March 1991; see the T in ChatGPT)

★ Deep Learning by Self-Supervised Pre-Training. Distilling NNs (Schmidhuber, April 1991; see the P in ChatGPT)

★ Experiments with Pre-Training; Analysis of Vanishing/Exploding Gradients, Roots of Long Short-Term Memory / Highway Nets / ResNets (Hochreiter, June 1991, further developed 1999-2015 with other students of Schmidhuber)

★ LSTM journal paper (1997, most cited AI paper of the 20th century)

★ xLSTM (Hochreiter, 2024)

★ Reinforcement Learning Prompt Engineer for Abstract Reasoning and Planning (Schmidhuber 2015)

★ Mindstorms in Natural Language-Based Societies of Mind (2023 paper by Schmidhuber's team)

https://arxiv.org/abs/2305.17066

★ Bremermann's physical limit of computation (1982)

EXTERNAL LINKS

CogX 2018 - Professor Juergen Schmidhuber

https://www.youtube.com/watch?v=17shdT9-wuA

Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability (Neural Networks, 1997)

https://sferics.idsia.ch/pub/juergen/loconet.pdf

The paradox at the heart of mathematics: Gödel's Incompleteness Theorem - Marcus du Sautoy

https://www.youtube.com/watch?v=I4pQbo5MQOs

(Refs truncated, full version on YT VD)

  continue reading

185集单集

Alle episoder

×
 
Loading …

欢迎使用Player FM

Player FM正在网上搜索高质量的播客,以便您现在享受。它是最好的播客应用程序,适用于安卓、iPhone和网络。注册以跨设备同步订阅。

 

快速参考指南