Greg Badros

During yesterday's LibreQoS APNIC webinar, I posed the question: won't more bandwidth solve these problems? As Herbert Wolverson said, yes, more bandwidth is good, but, still these problems remain if queueing is done incorrectly. Here's my take on why bandwidth alone is not as good as bandwidth + good queueing policies: Generally it was believed that 'data is important, so don't throw it away; hang on to it and send it later'. In practice, this has proven to be suboptimal as two issues may emerge: 1) latency increases for some flows due to heavy demand from other flows using the same bottleneck link, 2) even a single flow can have excessive latency due to aspects of typical TCP behavior (referred to bufferbloat) when the buffers grew large enough that the data being buffered was retransmitted anyway. It turns out that not all data is equally important. Active Queue Management is the art of deciding priorities, both in deciding what data to throw away, but also in allowing some later arriving data to be transmitted ahead of data in another connection that arrived before it. These problems have been studied, and good solutions have been found by using certain queueing policies in routers and switches, referred to as “fq_codel’ and “cake”. These track the different flows not by classifying the data, but by watching the behavior. Flows that send relatively little data (DNS lookups), or at a measured pace (video chat) have priority over flows that send a lot of data as quickly as possible (App and System updates, Video and ISO downloads).

10

5 Comments

I like this idea of "technical taste". It gets into where software engineering is as much creative art as it is science. Four great takeaways (IMHO) for software devs in this era in particular: 1. Aspiring engineers should cultivate a sense of curiosity, experiment with different tools and technologies, and embrace a mindset of continuous learning. 2. AI has the potential to streamline processes and enhance productivity for engineers, but it may also lead to disruptions in traditional software development workflows. 3. Developing technical taste and judgment is essential for making informed decisions about which technologies and approaches to pursue. 4. Collaboration and open-mindedness are key to leveraging the full potential of AI and staying ahead of technological advancements. I also really liked these two quotes from Sam Schillace: 1. "The right time to do something is when you have that feeling in the pit of your stomach that's like, 'oh, this is a great idea and it's going to suck to build because nothing's ready yet.'" 2. "Technical taste is like, 'how well have you consolidated that set of experiences and heuristics into judgment that you can apply accurately when you see new things?'" 3. "It may be the case that very small teams can do very large projects, or like we were talking about before, it may be the case we're just going to get really ambitious about what we try to do with the same size of teams, which is kind of where I would put my money." Great job on continuing to put out great episodes, Brett Berson and team First Round Capital! #ai #engineering #softwaredevelopment

5

Worth reading: Generative AI is not going to build your engineering team for you. So, so good - peppered with gem quotes and observations- chuckle. https://lnkd.in/gZBKtU4X Thanks to Eric Brooke for sharing.

16

Introducing the Alinia AI Alignment Platform. From their announcement: "Alinia AI enables companies to evaluate the performance of their GenAI applications and obtain evidence on how aligned these are to their values, policies, and business preferences. Following the performance assessment, Alinia AI recommends the best next actions to further guide and control LLM output and improve alignment."

10

5 Comments

Batch processing for LLM inference activity: "One sign of that is the rise of batch processing. Today, popular consumer AI products like ChatGPT and Perplexity provide users with near instantaneous responses, otherwise known as real-time inference. However, businesses don't always need immediate responses and are often willing to wait hours, days, or even weeks for responses, as long as they don't have to shell out as much money. Founders of cloud and inference providers have told me there's growing pressure from business customers for this kind of flexibility, otherwise known as batch processing." -- Nathaniel Whittemore on "The AI Daily Brief" (with much deserved shout-out to his inspo from Stephanie Palazzolo at The Information) I fall into this bucket as we have a lot of backfilling to do as we're building out the underlying data in our service, and some of that could just as well be done in the wee hours of the night. The new stuff flowing in I want processed ASAFP, but these backfills—I'd mos def take more cost-efficient delayed batch processing. There's also use cases like needing (near) real-time inference for the first inference pass for timeliness / contemporaneousness value, but then a second pass slow-time inference for wider classification exercises over an ever-growing data corpus. I could see us mixing Sonnet & Haiku models in both cases. Just a heads up to Anthropic Mike Krieger as you contemplate this, along with Amazon Web Services (AWS) Jeff Barr Mark Roy Swami Sivasubramanian on the Bedrock side of things...and I don't mean the "batch" mode that doesn't seem to have any real value other than helping AWS manage its own rate limit challenges. (DeepCast is using Bedrock.)

10

2 Comments

The afternoon Keynote @ Stripe Sessions was Fireside Chat between Patrick Collison (CEO of Stripe) and Jensen Huang (CEO of NVIDIA). Patrick was an excellent moderator — asked Jensen really probing, nuanced questions that Jensen answered with his usual sharp wit and unique (and impressively well rationalized) perspective. A couple of quotes among many that stuck with me: - “Current CPU installed market is $1 trillion of hardware, all of which will be upgraded to GPUs over the next 5 years. And I would estimate about another $1 trillion will be spent on top of that during the same time period as companies try to do more with AI. Add those together — and I think at least $2 trillion will be spent on GPUs in the next 5 years.” - “I like to go after Zero-Billion Dollar Markets — nonexistent markets that I envision & can rationalize well, but will be very hard, expensive & time-consuming to achieve.”

1

📈 AI Startup Sierra Expands HQ in San Francisco (4K to 41K SF) 🌁 Artificial intelligence firm Sierra is making a bold move by relocating to a larger office within San Francisco. The startup has signed a lease for a 41,104-square-foot space at 235 Second St., significantly upsizing from its current 4,000-square-foot office. 🏢 This move to San Francisco's South of Market neighborhood, where office vacancy rates have started to decline for the first time since 2020, is a positive sign for the city's hard-hit office market. The area's office vacancy rate, though still high at 28.8%, is slightly lower than six months ago. 📉 The deal contributes to the increase in leasing activity in San Francisco, which is up by about 1 million square feet compared to last year. This uptick is driven by the region's reputation as a leading national hub for AI companies. 💼 Sierra joins other AI firms expanding in the South of Market area, including Scale AI, which signed the city's largest office lease of 2024 with a 178,000-square-foot deal. So far this year, office tenants have signed leases totaling 750,000 square feet in South of Market, a significant increase from last year's 370,000 square feet. 🚀 Co-founded in 2023 by former Salesforce CEO Bret Taylor and former Google executive Clay Bavor, Sierra raised $85 million earlier this year to develop a customer support AI tool for businesses. The firm is set to relocate from its current space at 215 Second St. by next month. San Francisco's reputation as an AI hub, bolstered by its talent pool and six major AI research institutions, continues to attract companies. Recent significant deals include OpenAI’s sublease of over 400,000 square feet of Uber’s former headquarters and Anthropic’s takeover of Slack's former headquarters. These developments reflect a resilient and adaptive real estate market. As a commercial real estate professional, I’m excited to see how these trends shape the future of office space in our vibrant city.

21

Science-fiction is inherently speculative. Author George #Orwell of ‘#1984’ predicted concepts such as doublethink, little earbuds and society’s obsession with the black mirror screens that dominate our lives — and according to the BBC, novelist and short story author JG Ballard, of “High-Rise” and “Crash” fame, has predicted the current tool and cultural fixation of #GenerativeAI in the modern society — only he didn’t just speculate them within a story, he is actually one of the first use-cases of a generative AI used in a creative context — Ballard was so fascinated with the growing technological advancement of computers that he used one to compose poems, essentially the precursor to what we have now with ChatGPT. Today, the debate about the creative use of generative AI and its value on creative industries is an extremely fraught and controversial topic, but if controlled in a space that doesn’t replace human labor or try to equate itself to traditional art forms, it is interesting to see what computers can do and how they are creatively limited in comparison. Ballard’s two poems, The Yellow Back Novels and Machine Gun City are generated, and both include notes on how Ballard achieved producing them. Regardless of quality (Ballard himself probably wouldn’t describe these two poems as his best work), it is interesting to see primitive versions of the technology that we utilize on the daily today, and how these moral conversations that concern them would continuously evolve.

1

1 Comment

It’s THIS article (the second one in the post below): https://lnkd.in/dEdPxy62

2

Emily M. Bender does a great explanation in this article. I like how she uncovers the marketing term #AI with the real underlying concept of #automation. BTW, software automation was being done before the widespread use of ML. The bigger issue, as she rightly points out, is how it is hard to understand the bias that comes from training with data a system that is going to be used to make a decision.

4

1 Comment

What’s happening with generative AI in healthcare & life sciences is pretty exciting. Today, EvolutionaryScale launched their cutting-edge ESM3 model for researchers and scientists, and it’s available on Amazon Web Services (AWS). ESM3 has the remarkable ability to generate entirely new proteins that have never existed before, allowing for “programmable biology.” Models like ESM3 have the potential to significantly speed drug discovery timelines and lower costs. The open-source ESM3 is available today through Amazon SageMaker and AWS HealthOmics, with the full model family coming to these services and Amazon Bedrock soon. We look forward to making it easier for scientists everywhere to use GenAI for drug discovery and development. https://lnkd.in/gVrA_zhu

2,887

75 Comments

~40% of Frec’s customers are engineers working in tech, many of whom expect frequent and large capital gains. They also prefer to self-manage their money, which makes Frec an ideal platform. Meet Evan: an engineering leader who upgraded from investing in ETFs to direct indexing. Direct indexing appealed to Evan because it offers comparable returns to his ETFs while saving him money on advisory and ETF fees. “ETFs are true and tested, and giving folks a different way of investing in the market while saving more money is really appealing.” He’s used many other robo-advisors and other fintech tools, but “Frec seems to maximize tax loss harvesting opportunities better than any other product that I’ve used.” He also likes Frec's transparent fee structure.  “I feel like with a lot of other companies, that’s muddled in with the cost of investments,” he told us, “and you don’t really know how much you’re paying or how much you’re not saving as a result. I think Frec does a really good job of laying all of that out and giving you a true understanding of the cost of these investment products.” Read more about his Frec journey here ���

45

2 Comments

Stability AI, a startup that rode the boom in generative artificial intelligence to a $1 billion valuation before experiencing financial trouble and employee defections, has a new CEO. Prem Akkaraju, former CEO of visual effects company Weta Digital, will step into the role. The 51-year-old Akkaraju is part of a group of investors including former Facebook President Sean Parker that has stepped in to save Stability with a cash infusion that could result in a lower valuation for the firm, which develops products that use Stable Diffusion, an open-source AI image generator. The new funding will likely shrink the stakes of some existing investors, who have collectively contributed more than $100 million. https://lnkd.in/egevkxv3 #artificialintelligence #ai #stabilityai #venturecapital #fundraising #startups #stablediffusion

3

1 Comment

Multi-agent systems have been steadily gaining adoption, but it’s important to understand that they aren’t a blanket solution.  In fact, they introduce a range of complexities around coordination, consistency, and coherence that single-agent setups avoid. In my conversation with Chi Wang, the creator of AutoGen, he underscored the importance of matching your product architecture to the problem you’re trying to solve. If your use case involves narrowly defined tasks, a lone agent may be the simpler, more efficient choice. Other factors to consider before making the single agent versus multi agent decision? Response speed, decision-making frequency, inter-agent communication needs, latency, and bandwidth. Check out my full conversation with Chi here:

39

9 Comments

Don't fight against AI. Help it to get latest info. 👇 is the talk I gave last week regarding Real-Time #RAG in Zilliz's Unstructured Data and #LLM meetup. "Talk is cheap, show me the code" This blog shows the key data flow and code blocks. The full example is available at https://lnkd.in/gUj349hh How it works? Input: Stream stories and comments from #HackerNews API. Pre-process: Retrieve updates and filter for stories/comments via #Bytewax. Retrieve Content: Download the html and parse it into usable text. Thanks to the awesome #Unstructured.io. Vectorize: Create an embedding or list of embeddings for text using #HuggingFace #Transformers. Output: Write the JSON document with #embeddings and other fields to a local #Timeplus #Proton server, into the data streams. Filter/Route: Use streaming SQL to filter/transform JSON data and send output to a local #Kafka server. Forward: Use #Milvus plugin for #KafkaConnect to forward the data to #Zilliz Cloud. Query: Build a #streamlit web application to query the live data with the same embedding model.

18

1 Comment

Thanks Joanne Chen and Jaya G. from Foundation Capital for the nod! Great to be in the cohort of top notch AI companies. Observability data is untapped goldmine of information, but when the time comes (in the middle of incident), it still requires "expert" skills to find the needle in the haystack. We at Deductive AI are democratizing the expertise to everyone in the organization.

34

4 Comments

Here's a needed reminder (and warning) from FinTech Futures about how commonplace #deepfake scams are becoming as large language models (#LLMs) and #AI tech improves. AI-generation is becoming sophisticated at replicating human-like speech. Phone scams, fraudulent videos, and other types of scams are going to become more convincing to those who are unprepared. This article also highlights FinVolution, a fintech platform that has a focus on anti-fraud technologies in the Asian region. Right now, algorithm scientists are trying to spot #hoaxes by spotting the nuances and imperfections that come with AI voice cloning technology, basically digital forensics to help customers identify the artificial. I do truly mean it when I stress that AI, like other forms of technology, is ideologically neutral. In the same way that plenty of smart innovators are utilizing AI for good, there are also bad faith actors misusing AI for their own nefarious purposes. So, even though we live in an exciting time for generative AI, lets also remember to practice caution and use our better judgement to protect ourselves from #scams, #misinformation, and #fraud caused by deepfake technology.

12

5 Comments

Apple’s Private Cloud Compute and the business of privacy With Private Cloud Compute, Apple has gone to extraordinary lengths to safeguard user privacy with AI-related tasks. In doing so, Apple has surely also incurred an extraordinary expense. How can Apple justify the cost of providing PCC if it doesn’t directly monetize its Apple Intelligence features? Either through the improved brand perception captured in a broader privacy narrative or by monetizing PCC with other non-AI use cases. While the narrative explanation is clear, the more interesting theory about Apple’s motivation in building PCC is that it can be expanded to other use cases that can be directly monetized. The first alternative use case that springs to mind is advertising attribution. PCC sounds conceptually similar to a data clean room, albeit far more sophisticated than most implementations, which merely use IP addresses and other identifiers to match publisher-observed ad interactions with advertiser-observed conversions. While the PCC nodes seem hardware-optimized for processing AI-related tasks, other nodes could be built that aren’t.

19

3 Comments