Marc Cohen

The OP is wrong about the count, I think they are counting all models in HF as LLMs. Because there are only 700K+ total models in HF. But it sure begs the question why ~100K text generation models (if not LLMs). For perspective sentence-transformer library based models are only ~7K Related reads ⭕️ Elon’s idiot index for LLMs: https://lnkd.in/eWsbM45J ⭕️ How open is your LLM: https://lnkd.in/eY6JE7Ae

15

3 Comments

I think this news did not get as much attention as it deserves More and more models , more and more different inference and training software libraries are available for chips alternating to NVIDIA. It can create an interesting dynamics Chip capabilities are part of the picture. Software libraries , packages to make this chip easily available for tasks is another very important part. I see a lot of work to make alternative chips usable

4

1 Comment

Five Codec-Related Techniques to Cut Bandwidth Costs - Try Per-Title Encoding This article explores five strategies for reducing bandwidth costs in video streaming, perfect for engineers and strategists looking to enhance streaming efficiency. Suggestion: Per-Title Encoding Implement per-title encoding to optimize bitrate ladders uniquely for each video, potentially saving substantial bandwidth. While many larger companies develop their own technology, smaller companies can achieve the same benefit by buying an encoder or using a cloud service with per-title encoding.

14

We are getting used to these days with frequent news on LLMs getting bigger and better, every now and then some foundation model is coming up with their next best thing, with a digit added to the model name, even in that noisy world – this week’s announcement of Meta’s Llama 3 has more than something to stand out. There has been a lot of talk since the introduction of Llama 3 on the 18th of April around the usual metrics and evaluations including multi-shot MMLU, ARC-Challenge etc. context window, the pretraining with 15T tokens and many others. For me, there are 3 things which stood out: 🔹 Open Source. Even a year back open source foundation models were not taken seriously beyond just experimentations. 🔹 Looking beyond English language data sets for pretraining. Llama 3 is pretrained using 30 different languages, its 5% of the overall dataset and the performance of these languages are not as par yet, but this is a good start. 🔹 A stress on Security and Safety. New threat vectors are getting introduced with the evolution of the LLMs, and also with the integration of the LLMs with other elements – like delegating some tasks to code interpreters for logic and mathematical reasoning. It is refreshing to see the attempts to look into the security elements, including protection from Prompt injection attacks and also Code interpreter abuse. Also, the attempt to introduce new benchmarks for evaluating the security protections – CyberSecEval 2. https://lnkd.in/eH84MPj2

16

Quick tip - DDL for vector indexes https://lnkd.in/gwtNtE_r

38

There is some confusion in the industry about causal reasoning versus other methods involved in observability. It's nuanced, but also clearly differentiated. If you want to see it in action, here's a blog on it https://lnkd.in/dk9xn33K

7

1 Comment

Although some #Kakfa vendors are considering S3 Express One Zone for write caching, a model developed by my colleague Jack Vanlightly suggests this works only in a few cases. And with cloud vendors like Microsoft reducing or eliminating cross availability zone networking charges, traditional Kafka replicated logs still work well for many workloads. https://lnkd.in/gHrNPfSZ How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out.

4

This is impressive, and local, and potentially private (or at least, more private than a native cloud only solution). If you haven't seen it yet, watch this quick video. #ai #chrome #gemini

1

ADATA Unveils Future-Ready Innovations at Computex 2024   Under the banner “Innovate Today, Embrace Tomorrow,” #ADATA Technology Co. Ltd., a global frontrunner in memory modules and flash memory, is set to join forces with its gaming subsidiary XPG and industrial-grade embedded storage brand ADATA Industrial at #ComputexTaipei2024 happening from June 4 to June 7. The event will spotlight ADATA’s commitment to spearheading innovation and sustainability, with product showcases in three key areas: “AI Computing,” “Sustainability and Innovation,” and “Immersive Reality.” Brand ambassador Mera of ADATA and XPG will grace the “AI Computing” segment with a surprise appearance. Here, ADATA will unveil the world’s premier AI gaming laptop, boasting an SSD with a staggering 24TB capacity and 96GB of DRAM. The event will also feature overclocked DDR5 8000 R-DIMM for handling extensive computing workloads and data processing, and a comprehensive suite of Express Card storage solutions that enhance AI mobile devices. In the “Sustainability and Innovation” section, ADATA will display a variety of high-performance products that continue to set the pace for sustainability trends. This includes the industry’s only all-encompassing innovative cooling solutions and a full spectrum of green storage products crafted from recycled materials. The “Immersive Reality” segment will introduce a new gaming handheld PC, the XPG NIA, alongside a complete range of gaming products and peripherals, and the industry’s most potent external SSD solutions and mobile peripherals. To make the exhibition more accessible to global consumers, ADATA will concurrently launch its online COMPUTEX event from June 4, allowing consumers to explore ADATA’s extensive range of industry-leading products. FAQs What is the theme for ADATA at Computex 2024? The theme for ADATA at Computex 2024 is “Innovate Today, Embrace Tomorrow.” What are the three major categories of products that ADATA will present at the show? ADATA will present products in three major categories: “AI Computing,” “Sustainability and Innovation,” and “Immersive Reality.” What is the capacity of the SSD in the AI gaming laptop that ADATA will showcase? The AI gaming laptop that ADATA will showcase is equipped with an SSD of up to 24TB. What are some of the high-performance products that ADATA will display in the “Sustainability and Innovation” section? ADATA will display various high-performance products that continue to lead sustainability trends, including the industry’s only comprehensive innovative cooling solutions and a full range of green storage products made of recycled materials. What new gaming device will be featured in the “Immersive Reality” section? The “Immersive Reality” section will feature a new gaming handheld PC, the XPG NIA. via APPSGADGET:https://lnkd.in/dgUeKKfE June 03, 2024 at 08:35PM

1

LLMs: On Soundness and Completeness of a Probabilistic Test Oracle Dear Testers, Imagine we have a Large Language Model (LLM) that generates responses to prompts. We also have a perfect ground truth oracle G, which always knows the right answer. Probabilistic Soundness We want to make sure that if our test oracle says the LLM's answer is correct, it's actually correct most of the time (better than a coin flip plus a little extra) Example 1. Suppose the LLM generates the answer "Paris is the capital of France." 2. The test oracle O evaluates this answer and says it is correct. 3. We want the probability that O is right to be greater than 1/2 + a little extra, as confirmed by G. 4. If this condition holds, O is considered to be sound. Probabilistic Completeness We also want to ensure that if the LLM's answer is truly correct according to the ground truth oracle G, our test oracle O will recognize it as correct more often than just guessing randomly plus that extra bit. Example 1. Suppose the LLM generates the answer "Paris is the capital of France," which the ground truth oracle G confirms as correct. 2. We want our test oracle O to recognize this as correct more often than 1/2 + a little extra. 3. If this condition holds, O is considered to be complete. The Problem The Ground Truth Oracle in itself is a myth in most contexts. The O in reality is - plural: Os. So, if we as testers think about automating testing of LLMs, it is a very intricate challenge. The challenge still remains if we test LLMs as humans as well because we prefer determinism as per our nature. The "a little extra" is an equally bigger problem. How much extra is good enough? Testing of LLMs is a big, interesting challenge before us as a community. We need to re-look at test design as well as the key issue - The Oracle Problem. #Testing #BackToBasics #TestOracles #LLMs

18

32 Comments

Building on the foundational insights from Part 1, we dive deeper into the practical applications of Knowledge Graphs (KGs) and Retrieval-Augmented Generation (RAG) in Part 2. Through a series of industry-specific examples, we explore how the KG-RAG model revolutionizes decision-making across various sectors. 𝗞𝗲𝘆 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀 𝗳𝗿𝗼𝗺 𝗣𝗮𝗿𝘁 𝟮: I provide a few practical examples across various functional entities such as: 🔧 Product Development 📈 Marketing 🛡️ Compliance 🤝 Customer Support 📦 Supply Chain Optimization ⚠️ Risk Management We also get into a deep-dive of a possible 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝘀𝗲𝗹𝗲𝗰𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗽𝗲𝗿𝗳𝗲𝗰𝘁 𝗱𝗶𝘀𝗽𝗹𝗮𝘆 𝗳𝗼𝗿 𝗮 𝘀𝗺𝗮𝗿𝘁 𝗱𝗲𝘃𝗶𝗰𝗲. This scenario illustrated how KG-RAG can navigate complex data relationships, from cost and durability to touch sensitivity and vendor reliability, ensuring informed and balanced decision-making. The post below was created using the LLMs, from creating the base outline to every detail. Please note that the blog is based on information provided by 2 LLMs. While it was a fun and terrific experience, I cannot guarantee that everything here is accurate. However, I do believe it tracks directionally :) Feel free to read the full blog post for an in-depth look at these transformative applications and how KG-RAG can benefit your industry. https://lnkd.in/gjuziwsv #KnowledgeGraphs #ArtificialIntelligence #MachineLearning #DataAnalytics #DecisionMaking #ProductDevelopment #MarketingStrategy #Compliance #CustomerSupport #SupplyChainManagement #RiskManagement #TechInnovation #BusinessIntelligence #SmartDevices #FutureOfWork Matt Sobel Kevin McFarland - a quick shout out to Writer as well!

24

2 Comments

Whisper on NPU, LLMs on iGPU, document embeddings on CPU! Full use of #AIPC through #OpenVINO to implement scenarios like chat with documents and voice guided assistant that generates code. Everything on our new Lunar Lake platform that was just announced. Pipelines are running locally, so documents and queries that you use never exposed to cloud. No need for network connection to use it, working from airplane unlocked. #SLMs that run locally is very promising direction in my opinion. We should not expect miracles in terms of actual knowledge from those models, #RAG-like pipelines will drive knowledge based context for model to analyze and respond. SLMs should be more about skill, not actual knowledge. https://lnkd.in/d27MDvPd Guy Boudoukh, Moshe Wasserblat, Ekaterina Aidova and team, great work on putting it together!

58

TransformerFAM: Feedback attention is working memory (Google, April 2024) Paper: https://lnkd.in/gw3Z723R Abstract: "While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length."

6

Kolmogorv-Arnold Networks (KANs) are a new type of network that was released from research last week. They different from Multilayer Perceptrons (MLPs) by focusing on learning functions at the edges instead of the weights at the nodes. In the coming months here's what I expect to happen: - People will test how large you can make a KAN while maintaining the original findings of the paper - Someone will test and plot the relationship between the number of KAN parameters and MLP parameters - There will be many papers comparing transformer architectures using MLPs vs KANs and that will attempt to outline situations where one works better than another - Someone will create a combined training-inference cost curve denoting the decision boundary for network size when using KANs vs MLPs pays off. Note that these are likely to be either much more scientific problems, much heavier on the inference equation, or on the smaller network side. - Someone will find a more efficient way to train KANs https://lnkd.in/eNTsNixv

4

Transformer-based LLMs have an inherent performance bottleneck. SSMs like Mamba provide an alternative LLM architecture that could work well when transformer-based LLMs struggle to scale with very large context. I've been intrigued by Mamba since it was launched back in March. Finally getting around to reading a bit more about it in this article from The Gradient: https://lnkd.in/ebVtWQPv

12