Lifeboat Foundation

W/ Andrej Karpathy

The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We’ll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.

Continue reading “Let’s build the GPT Tokenizer” »

“With the world growing more crowded, the great powers strive to conquer other planets. The race is on. The interplanetary sea has been charted; the first caravelle of space is being constructed. Who will get there first? Who will be the new Columbus?” A robot probe is being readied to explore the secrets of the red planet, Mars. The only component lacking: a human brain. No body. Just the brain. It is needed to deal with unexpected crises in the cold, dark depths of space. The perfect volunteer is found in Colonel Barham, a brilliant but hot-tempered astronaut dying of leukemia. But all goes awry as, stripped of his mortal flesh, Barham — or rather his disembodied brain — is consumed with a newly-found power to control…or destroy. Project psychiatrist Major McKinnon (Grant Williams) diagnoses the brain as having delusions of grandeur…but, just perhaps, Col. Barham has achieved grandeur.

‘Inception’ is about to become a documentary — or are we all just dreaming?

Researchers at Osaka University have developed a groundbreaking flexible optical sensor that works even when crumpled. Using carbon nanotube photodetectors and wireless Bluetooth technology, this sensor enables non-invasive analysis and holds promise for advancements in imaging, wearable technology, and soft robotics. Credit: SciTechDaily.com.

Researchers at Osaka University have created a soft, pliable, and wireless optical sensor using carbon nanotubes and organic transistors on an ultra-thin polymer film. This innovation is poised to open new possibilities in imaging technologies and non-destructive analysis techniques.

Recent years have brought remarkable progress in imaging technology, ranging from high-speed optical sensors capable of capturing more than two million frames per second to compact, lensless cameras that can capture images with just a single pixel.

Reminder: It’s only been 1 YEAR since the Will Smith eating spaghetti videoOpenAI Sora: https://openai.com/sora#researchThumbnail character: BasedAFMKBHD Mer…

LoRA Land: 25 fine-tuned #Mistral 7b #LLM that outperform #gpt4 on task-specific applications ranging from sentiment detection to question answering.

Predibase is the fastest way to productionize open-source AI. It enables easy fine-tuning and serving of LLM or deep learning models on cloud infrastructure, promising scalability and cost-effectiveness and is currently used by Fortune 500 and high-growth companies. Try for free: https://predibase.com/free-trial.

Speaker’s Bio: Catherine (Katie) Schuman is a research scientist at Oak Ridge National Laboratory (ORNL). She received her Ph.D. in Computer Science from the University of Tennessee (UT) in 2015, where she completed her dissertation on the use of evolutionary algorithms to train spiking neural networks for neuromorphic systems. She is continuing her study of algorithms for neuromorphic computing at ORNL. Katie has an adjunct faculty appointment with the Department of Electrical Engineering and Computer Science at UT, where she co-leads the TENNLab neuromorphic computing research group. Katie received the U.S. Department of Energy Early Career Award in 2019.

Talk Abstract: Neuromorphic computing is a popular technology for the future of computing. Much of the focus in neuromorphic computing research and development has focused on new architectures, devices, and materials, rather than in the software, algorithms, and applications of these systems. In this talk, I will overview the field of neuromorphic from the computer science perspective. I will give an introduction to spiking neural networks, as well as some of the most common algorithms used in the field. Finally, I will discuss the potential for using neuromorphic systems in real-world applications from scientific data analysis to autonomous vehicles.

Continue reading “Neuromorphic Computing from the Computer Science Perspective: Algorithms and Applications” »

Topological Deep Learning (TDL) is gaining traction for its ability to capture higher-order interactions beyond the pairwise structure of #graphs, using tools from #algebraic #topology, especially combinatorial topological spaces.

How combinatorial topological spaces can be used to promote a paradigm shift from inferring pairwise to multiway latent relationships in data.

Several problems in machine learning call for methods able to infer and exploit multi-way, higher-order relationships hidden in the data. We propose the new beyond-graph paradigm of Latent Topology Inference, which aims to learn latent higher-order combinatorial topological spaces describing multi-way interactions among data points. To make Latent Topology Inference implementable, we introduce the Differentiable Cell Complex Module, a novel learnable function able to infer a latent cell complex to improve the downstream task.

Continue reading “Latent Topology Inference” »

Along with climbing homelessness and other societal woes globally this is the time for transhuman ideals to emerge to save lives. We could automate all work and get universal basic income with AI to work for us.

About 61% of Americans are living paycheck to paycheck, an issue that impacts both low-wage and high-income families alike, according to new research from LendingClub.

Low-wage earners are most likely to live paycheck to paycheck, with almost 8 in 10 consumers earning less than $50,000 a year unable to cover their future bills until their next paycheck arrives. Yet even 4 in 10 high-income Americans, or those earning more than $100,000, say they’re in the same position, the research found.

Continue reading “More than 60% of Americans are living paycheck to paycheck. Here’s what researchers say is to blame” »