[QA] Using Attention Sinks To Identify And Evaluate Dormant Heads In Pretrained LLMs Arxiv Papers podcast

7:29

This paper challenges the necessity of lengthy reasoning processes in LLMs, showing that simple prompting (NoThinking) can outperform traditional methods in various reasoning tasks, especially in low-budget scenarios. https://arxiv.org/abs//2504.09858 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Reasoning Models Can Be Effective Without Thinking 20:05

19 hours之前20:05

20:05

This paper challenges the necessity of lengthy reasoning processes in LLMs, showing that simple prompting (NoThinking) can outperform traditional methods in various reasoning tasks, especially in low-budget scenarios. https://arxiv.org/abs//2504.09858 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce 8:27

19 hours之前8:27

8:27

This paper analyzes GRPO in reinforcement learning for language models, revealing that a simple rejection sampling method, RAFT, performs competitively and suggesting improvements for future reward-based training approaches. https://arxiv.org/abs//2504.11343 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce 14:38

19 hours之前14:38

14:38

This paper analyzes GRPO in reinforcement learning for language models, revealing that a simple rejection sampling method, RAFT, performs competitively and suggesting improvements for future reward-based training approaches. https://arxiv.org/abs//2504.11343 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training 7:14

2天之前7:14

7:14

https://arxiv.org/abs//2504.13161 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers

A

Arxiv Papers

1
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training 20:35

2天之前20:35

20:35

https://arxiv.org/abs//2504.13161 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers

A

Arxiv Papers

1
[QA] Antidistillation Sampling 7:21

2天之前7:21

7:21

Antidistillation sampling modifies token probability distributions to weaken reasoning traces for model distillation, enhancing model security while maintaining performance. https://arxiv.org/abs//2504.13146 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Antidistillation Sampling 10:44

2天之前10:44

10:44

Antidistillation sampling modifies token probability distributions to weaken reasoning traces for model distillation, enhancing model security while maintaining performance. https://arxiv.org/abs//2504.13146 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] Position: The Most Expensive Part of an LLM should be its Training Data 7:16

3天之前7:16

7:16

This paper argues that compensating human labor for training data is the largest cost in developing Large Language Models, significantly exceeding model training expenses, and suggests fairer practices for the future. https://arxiv.org/abs//2504.12427 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Position: The Most Expensive Part of an LLM should be its Training Data 20:05

3天之前20:05

20:05

This paper argues that compensating human labor for training data is the largest cost in developing Large Language Models, significantly exceeding model training expenses, and suggests fairer practices for the future. https://arxiv.org/abs//2504.12427 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] Activated LoRA: Fine-tuned LLMs for Intrinsics 8:16

3天之前8:16

8:16

Activated LoRA (aLoRA) enhances LoRA by adapting weights only for relevant tokens, allowing instant activation without recomputing the KV cache, improving efficiency in multiturn settings. https://arxiv.org/abs//2504.12397 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Activated LoRA: Fine-tuned LLMs for Intrinsics 18:55

3天之前18:55

18:55

Activated LoRA (aLoRA) enhances LoRA by adapting weights only for relevant tokens, allowing instant activation without recomputing the KV cache, improving efficiency in multiturn settings. https://arxiv.org/abs//2504.12397 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] COLORBENCH: Can VLMs See and Understand the Colorful World? 7:49

3天之前7:49

7:49

The paper presents COLORBENCH, a benchmark to evaluate vision-language models' color understanding, revealing limitations and emphasizing the need for improved color comprehension in multimodal AI. https://arxiv.org/abs//2504.10514 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
COLORBENCH: Can VLMs See and Understand the Colorful World? 20:40

3天之前20:40

20:40

The paper presents COLORBENCH, a benchmark to evaluate vision-language models' color understanding, revealing limitations and emphasizing the need for improved color comprehension in multimodal AI. https://arxiv.org/abs//2504.10514 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs 8:33

3天之前8:33

8:33

https://arxiv.org/abs//2504.11536 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers

A

Arxiv Papers

1
ReTool: Reinforcement Learning for Strategic Tool Use in LLMs 14:57

3天之前14:57

14:57

https://arxiv.org/abs//2504.11536 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers

A

Arxiv Papers

1
[QA] Looking beyond the next token 7:22

4天之前7:22

7:22

The paper presents TRELAWNEY, a method for rearranging training data to improve causal language models' performance in planning and reasoning without altering architecture, enhancing goal generation capabilities. https://arxiv.org/abs//2504.11336 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Looking beyond the next token 16:58

4天之前16:58

16:58

The paper presents TRELAWNEY, a method for rearranging training data to improve causal language models' performance in planning and reasoning without altering architecture, enhancing goal generation capabilities. https://arxiv.org/abs//2504.11336 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] How to Predict Best Pretraining Data with Small Experiments 8:16

4天之前8:16

8:16

The paper introduces DATADECIDE, a suite for evaluating data selection methods, revealing that small-scale model rankings effectively predict larger model performance, enhancing cost-efficient pretraining decisions. https://arxiv.org/abs//2504.11393 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
How to Predict Best Pretraining Data with Small Experiments 20:22

4天之前20:22

20:22

The paper introduces DATADECIDE, a suite for evaluating data selection methods, revealing that small-scale model rankings effectively predict larger model performance, enhancing cost-efficient pretraining decisions. https://arxiv.org/abs//2504.11393 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability 7:18

6天之前7:18

7:18

This study evaluates OpenAI's GPT-4o, revealing limitations in semantic synthesis, instruction adherence, and reasoning, challenging assumptions about its multimodal capabilities and calling for improved benchmarks and training strategies. https://arxiv.org/abs//2504.08003 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability 7:07

6天之前7:07

7:07

This study evaluates OpenAI's GPT-4o, revealing limitations in semantic synthesis, instruction adherence, and reasoning, challenging assumptions about its multimodal capabilities and calling for improved benchmarks and training strategies. https://arxiv.org/abs//2504.08003 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training 7:39

6天之前7:39

7:39

This paper introduces a distribution-level curriculum learning framework for RL-based post-training of LLMs, enhancing reasoning capabilities by adaptively scheduling training across diverse data distributions. https://arxiv.org/abs//2504.09710 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training 10:11

6天之前10:11

10:11

This paper introduces a distribution-level curriculum learning framework for RL-based post-training of LLMs, enhancing reasoning capabilities by adaptively scheduling training across diverse data distributions. https://arxiv.org/abs//2504.09710 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] Steering CLIP's vision transformer with sparse autoencoders 8:11

6天之前8:11

8:11

This study explores sparse autoencoders in vision models, revealing unique processing patterns and enhancing steerability, leading to improved performance in vision disentanglement tasks and defense strategies. https://arxiv.org/abs//2504.08729 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Steering CLIP's vision transformer with sparse autoencoders 17:53

6天之前17:53

17:53

This study explores sparse autoencoders in vision models, revealing unique processing patterns and enhancing steerability, leading to improved performance in vision disentanglement tasks and defense strategies. https://arxiv.org/abs//2504.08729 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning 7:58

6天之前7:58

7:58

Genius is an unsupervised self-training framework that enhances LLM reasoning without external supervision, using stepwise foresight re-sampling and advantage-calibrated optimization to improve performance. https://arxiv.org/abs//2504.08672 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning 18:11

6天之前18:11

18:11

Genius is an unsupervised self-training framework that enhances LLM reasoning without external supervision, using stepwise foresight re-sampling and advantage-calibrated optimization to improve performance. https://arxiv.org/abs//2504.08672 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
[QA] Rethinking Reflection in Pre-Training 8:18

8天之前8:18

8:18

The study reveals that language models develop self-correcting abilities during pre-training, enhancing their problem-solving skills, as demonstrated by the OLMo-2-7B model's performance on self-reflection tasks. https://arxiv.org/abs//2504.04022 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers…

A

Arxiv Papers

1
Rethinking Reflection in Pre-Training 17:47

8天之前17:47