Advances in Deep Learning: Transformers

Prof. Yossi Keshet

Sep 22-23, 2024

 

Please leave your information here, and we will get back to you shortly when registration opens.

  The Transformer architecture has emerged as a groundbreaking deep learning model, revolutionizing various domains with its powerful representation of sequential data. This course delves into the fascinating world of Transformers, exploring their fundamental concepts, theoretical underpinnings, and practical applications.   We begin by thoroughly examining the Transformer model itself, unpacking its innovative self-attention mechanism and its ability to capture long-range dependencies efficiently. This theoretical foundation will provide a solid understanding of the architectural principles that have propelled Transformers to the forefront of modern deep learning.   Building upon this knowledge, we will explore the transformative impact of Transformers across multiple fields. In natural language processing (NLP), we will study how Transformers have enabled the development of large language models like BERT and ChatGPT, pushing the boundaries of text generation, understanding, and analysis.   Furthermore, we will delve into the realm of automatic speech recognition (ASR), where Transformer-based models such as wav2vec, HuBERT, and Whisper have achieved state-of-the-art performance, revolutionizing the way we interact with spoken language.   Extending our exploration to computer vision, we will examine how Transformers have been adapted to handle visual data, leading to groundbreaking models that have reshaped image and video understanding tasks.   Throughout the course, we will critically analyze the latest research developments, discussing theoretical advancements, practical applications, and the potential future directions of Transformer architectures. By the end of this comprehensive journey, you will possess a deep understanding of Transformers, equipping you with the knowledge and skills to harness their power in your own research or industry projects.  

• Introduction
• The roots of the Transformer architecture and how it was developed: RNN, LSTM, sequence-to-sequence, attention mechanism, LAS
• Transformers structure and the self-attention mechanism
• Components of Transformers and their usage
• What is the expressive power of Transformers?
• Reinforcement learning from human feedback (RLHF) and Direct Preference Optimization (DPO)
• Efficient Transformers: Flash Networks; RWKV
• In-context learning
• Prompts and the limitation of the alignment problem
• Transformers implementation in NLP
• Transformers Implementation in Speech
• Transformers Implementation in Vision

   

Prof. Ido Roll: Associate Professor and Senior Vice Provost for the Advancement of Learning and Teaching, Faculty of Education in Science and Technology and Faculty of Data and Decision Sciences Tuesday, June 18, 2024, 17:30-18:30, Faculty of Data and Decision Sciences, Bloomfield Building, 5th Floor, Room 526. Artificial intelligence tools have the potential to revolutionize higher education or become just another item on a long list of tools that couldn't break tradition. I will briefly present some specific developments in the field of artificial intelligence in higher education, followed by a discussion on the topic, both generally and at the Technion in particular. The only rules: no overgeneralizations (robots will kill/serve us all) and no excessive anecdotes (my niece wrote a Eurovision song in Swahili/Yiddish with ChatGPT). The meeting will be held in Hebrew Please register for the meeting here: הרשמה למפגשי בינה מלאכותית בחינוך, הטכניון (google.com)

בינה מלאכותית והחינוך הגבוה - לאן אנו צועדים?

פרופ' עדו רול: פרופסור חבר וסגן המשנה הבכיר לנשיא לקידום הלמידה וההוראה, הפקולטה לחינוך למדע וטכנולוגיה והפקולטה למדעי הנתונים וההחלטות

יום שלישי, 18.6.24, 17:30-18:30, הפקולטה למדעי הנתונים וההחלטות, בניין בלומפילד, קומה 5, חדר 526.

לכלי בינה מלאכותית פוטנציאל לעשות מהפיכה של ממש בחינוך הגבוה, או להיות עוד הזדמנות ברשימה ארוכה של כלים שלא יכלו למסורת. אציג בקצרה מספר התפתחויות ספציפיות לתחום של בינה מלאכותית בחינוך הגבוה ואז נערוך דיון בנושא באופן כללי ובטכניון בפרט. הכללים היחידים: בלי הכללות יתר (הרובוטים יהרגו/ישרתו את כולנו) ובלי יותר מדי אנקדוטות (אחיינית שלי כתבה שיר לאירוויזיון בסוואהילית/יידיש בעזרת צ׳אט).

On Monday, 27.5.2024, the Tech.AI Biomed conference, will be held in the Technion Visitors Center. Registration now open!  

It was mR Tk ! We tried to touch on the future, to understand how to understand the shared space and how best to organize technologically and humanly for the moment when we will live together - an autonomous machine alongside man.
more details

Today it was Prof. Yael Yaniv from the Faculty of Biomedical Engineering who spoke to us about the integration of AI in the medical world in general and in the field of cardiology in particular.
more details

Yoni left us inspired and motivated to explore the vast possibilities that AI brings to the field. Join us next week for another incredible meetup by Tech.AI where Professor Yael Yaniv will discuss the challenges of implementing artificial intelligence tools in medical practice, particularly in the field of cardiology.
more details

This talk was held in cooperation with Tech.AI as part of the Tech.AI on Stage series. A big thanks to our community that keeps on growing and joining our meetups and events!
more details

As part of the lecture series "Tech.AI on stage" . Tomorrow a fascinating lecture will be given by Dr. Chaim Baskin from the Faculty of Computer Science. Topic: Learning from Noisy Labeled Data Tomorrow, Monday 14.8 at 16:00.
more details

more details