Journeys in Middle-Earth of LLMs

About

Large Language Models (LLMs) have become a crucial part of AI research and applications. However, we often use them without deeply understanding their foundations. This journal club aims to explore LLMs from the ground up, covering seminal papers, key advancements, and their limitations.

Format

Each session lasts approximately 1 hour and 15 minutes, consisting of two presentations:

• Long Paper Presentation: 30-minute presentation followed by a 15-minute Q&A session. The audience is encouraged to read the paper beforehand and actively participate in discussions.
• Short Paper Presentation: A 15-minute presentation on a recent, relevant paper, followed by 10 minutes of discussion. The paper could be proposed by the speaker and it should be shared at least a week in advance.

Speakers may choose their presentation style, whether using slides, the paper itself, a blackboard, or a combination.

Schedule

Papers Proposals

• Transformers and BERT: For this session, I propose spending 15 minutes on Attention Is All You Need (which is probably already familiar) and dedicating 30 minutes to the new paper by LeCun, in order to compare the two papers.
  • Attention Is All You Need
  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  • Transformers without Normalization. This work demonstrates that Transformers without normalization can achieve the same or better performance using Dynamic Tanh operations.
• In-Context Learning (ICL): For this session, the goal is to first understand what ICL is, and then explore how the models use it.
  • A Survey on In-context Learning - Long presentation
  • Language Models are Few-Shot Learners (The seminal paper introducing the ICL terminology) - Long presentation
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? - Short presentation
  • Larger Language Models do In-Context Learning Differently - Short presentation
  • Lost in the Middle: How Language Models Use Long Contexts - Short presentation
• LLMs: For this session, depending on the selected papers, the focus could be on: (i) To Chain-of-Thought or not to Chain-of-Thought? (ii) What are Chain-of-Thought and Hypotheses-to-Theories?
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (CoT does help) - Long presentation
  • Scaling Laws for Neural Language Models (OpenAI) - Long presentation
  • Large Language Models Can Learn Rules (Google DeepMind: Hypotheses-to-theories) Long/Short presentation
  • Improving Language Understanding by Generative Pre-Training (GPT, OpenAI) - Long presentation
  • Chain of Thoughtlessness? An Analysis of CoT in Planning (CoT does not help) - Short presentation
• Argumentation/Explainability: For this session, the goal is to revisit the topic of argumentative AI and explore the applications of LLMs in this context.
  • Explanation in Artificial Intelligence: Insights from the Social Sciences - Long presentation
  • Argumentative review aggregation and dialogical explanations - Long presentation
  • Exploring the Potential of Large Language Models in Computational Argumentation - Short presentation
  • Performance analysis of large language models in the domain of legal argument mining - Short presentation
  • Large Concept Models: Language Modeling in a Sentence Representation Space - Short presentation
• Some additional papers that could be of interest for short presentations:
  • Ignore Previous Prompt: Attack Techniques for Language Models (Fooling LLMs)
  • Breach By A Thousand Leaks: Unsafe Information Leakage In Safe AI Responses (Fooling LLMs)

Best Speaker Award

The Best Speaker Award will be announced in December during the MARIANNE seminar. Stay tuned!