Arxiv Papers

Sparse Sinkhorn Token Translation (S2T2) improves text compression and inference in new domains by training tailored tokenizers and enabling effective token translation, enhancing performance in language models. https://arxiv.org/abs//2411.00593 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:08:13

Adapting Language Models via Token Translation

Duration:00:09:33

[QA] Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models

Specialized Sparse Autoencoders (SSAEs) enhance interpretability of foundation models by effectively capturing rare concepts, improving classification accuracy, and revealing insights into subdomain representations. https://arxiv.org/abs//2411.00743 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:08:29

Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models

Duration:00:26:54

[QA] Tokenformer: Rethinking Transformer Scaling with Tokenized Model Parameters

Tokenformer introduces a scalable architecture that enhances Transformers' efficiency by using token-parameter attention, allowing for incremental scaling without retraining, thus reducing computational costs significantly. https://arxiv.org/abs//2410.23168 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:07:51

Tokenformer: Rethinking Transformer Scaling with Tokenized Model Parameters

Duration:00:19:10

[QA] $100K or 100 Days: Trade-offs when Pre-Training with Academic Resources

This paper challenges the assumption that academic researchers can't pre-train models, providing benchmarks and insights on optimizing GPU resources for efficient model training. https://arxiv.org/abs//2410.23261 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:07:22

$100K or 100 Days: Trade-offs when Pre-Training with Academic Resources

Duration:00:16:51

[QA] What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective

11/1/2024

This study analyzes layer-wise gradients in LLMs, revealing that slow thinking enhances learning stability and response correctness, while fast thinking shows larger gradient variations. https://arxiv.org/abs//2410.23743 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:07:59

What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective

11/1/2024

Duration:00:15:27

[QA] Tokenformer: Rethinking Transformer Scaling with Tokenized Model Parameters

10/31/2024

Tokenformer introduces a scalable architecture that enhances Transformers' efficiency by treating model parameters as tokens, allowing for flexible scaling without retraining, significantly reducing computational costs. https://arxiv.org/abs//2410.23168 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:07:28

Tokenformer: Rethinking Transformer Scaling with Tokenized Model Parameters

10/31/2024

Duration:00:19:38

[QA] Where Do Large Learning Rates Lead Us?

This study investigates optimal initial learning rates for neural networks, finding a narrow range enhances generalization by locating high-quality minima and focusing on relevant features, unlike extreme rates. https://arxiv.org/abs//2410.22113 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:08:30

Where Do Large Learning Rates Lead Us?

Duration:00:28:43

[QA] Fourier Head: Helping Large Language Models Learn Complex Probability Distributions

The paper introduces a Fourier series-based neural network layer to improve continuous token modeling in decision-making and time series tasks, enhancing performance in various benchmarks. https://arxiv.org/abs//2410.22269 YouTube: https://www.youtube.com/@ArxivPapers TikTok: https://www.tiktok.com/@arxiv_papers Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016 Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers --- Support this podcast: https://podcasters.spotify.com/pod/show/arxiv-papers/support

Duration:00:07:10

Fourier Head: Helping Large Language Models Learn Complex Probability Distributions