Writing Your Own Attention-Based Transformers in Python:
Model General Sequences From Credit Scoring to Game Analysis

Rense Lange, Ph.D.
Workshop to be held January 9, 2024 at Lusofona Campo Grande, 376 1749-024 Lisboa, Portugal
Serious Gaming Conference etc
Cost: Free

 

This full-day workshop is designed for programmers and non-programmers alike, and it offers a deep dive into the world of transformers as embodied in models like ChatGPT, Llama, and the  HuggingFace modules. These transformers all draw inspiration from the seminal 2017 paper by Vaswani et al., which introduced "attention-based learning" focussing on natural language processing as a stream of words. However, instead of discussing yet another language-oriented transformer, this workshop aims to show how to harness transformers' capabilities for sequential applications in general. These applications span a wide Machine Learning (ML) spectrum, covering the analysis of moves in sequential games, general time-series predictions, bank loan repayment patterns, identifying patterns in physiological data, etc.

The workshop starts by offering a fairly comprehensive overview of the familiar Python computer language, aimed at enhancing participants’ proficiency. Not only does this enhance participants’ understanding of transformers, but it also enables them to develop their own transformer based applications later. Covered are topics such as data types, control mechanisms, the class concept and inheritance, and co-routines. Next, I introduce basic elements of Pytorch, which is quickly becoming the major ML neural net framework. Here I cover tensor manipulation, loss functions, softmax, backpropagation, and the utilization of test, training, and validation datasets. By executing instructive Python / Pytorchcode on their own computers, participants will gain hands-on experience with creating, running, and extending actual code. I believe that such experience is crucial to real learning.

All workshop materials, comprising functional Jupyter Notebooks and supplementary reading material, will be provided to participants without cost. To ensure a seamless learning experience, attendees are required to bring their own laptops equipped with Jupyter Notebook (Python V3.8 or higher) and an operational Pytorch installation (Version 2.0 or higher). Collaborative work is encouraged, allowing participants to team up with one or more partners. It is additionally recommended to install essential libraries such as Numpy, Pandas, Matplotlib, and scikit-learn. All aforementioned software is available for free download from the internet.