David Meyer

In case you missed it, we have now released the second issue of our Confluent Chronicles – a fun and entertaining take on the fundamentals of stream processing, Apache Flink, and Confluent! We had such great feedback on the first issue that we couldn’t just stop there – you can check out and download the latest edition here: https://lnkd.in/gEFxA7af Would love to hear what you think of it!   #ApacheFlink #ApacheKafka #Confluent

39

1 Comment

We are excited to announce the launch of Ursa engine, our Kafka API-compatible data streaming engine that is built on top of Lakehouse. Ursa is not something entirely new; it is built on our many years of experience building and operating #ApachePulsar and #ApacheKafka. To learn more about the #Ursa engine, I recommend our #PulsarSummit keynote presentation and announcement blog post. Here, I’d like to delve deeper into some of its features: 1. Making Kafka API a first-class citizen: In #ApachePulsar, we developed a unified messaging model combining queuing and streaming semantics into one protocol, aimed at enhancing data sharing between business applications. Although powerful and increasingly adopted by enterprises and unicorns, the requirement for application rewrites is impractical in today’s economy. Instead of building data gravity around a protocol, we open up access to the robust stream storage layer by extending support to the Kafka protocol, transitioning from a single protocol to a multi-protocol platform, facilitating greater flexibility and integration. 2. Achieve Stream-Table duality via Lakehouse Storage: Data streams commonly end up in data lakehouses, where integrating and maintaining them can be complex and resource-intensive. Ursa simplifies this by storing data streams as lakehouse tables in object storage. This integration not only streamlines operations but also allows seamless access to operational data for analytical processes. Thanks to its multi-modal storage engine which fosters data sharing across teams and departments. 3. Object storage as the primary storage solution: To better align with cloud economics, particularly for large workloads, Ursa makes ZooKeeper and BookKeeper optional and utilizes a cost-optimized write-ahead log based solely on object storage with leaderless and stateless brokers. This architecture leverages object storage for both data replication and storage, minimizing inter-AZ data transfers and reducing costs for latency-relaxed, high-volume workloads. By integrating these features at the multi-tenant level rather than at isolated cluster levels, Ursa eliminates costly data duplication and movement across clusters, allowing teams to map directly to tenants and be able to share data across teams and departments more efficiently. We believe that the future of data streaming platforms lies in being open and multi-faceted, fundamentally different from traditional data platforms. They are designed not just to store and process data, but to enhance organizational capabilities by fostering data sharing across the organization and integration with other systems. More and more data streaming platforms will look more like Ursa, with multi-protocol, multi-tenant, and multi-modal capabilities. Check out our announcement blog post about Ursa: https://lnkd.in/ghEKTYE6 #ApacheKafka #Lakehouse #DataStreaming

100

1 Comment

If you’re struggling to manage an exploding number of #APIs, I want to reassure you that you’re not alone. Our newly released 2024 State of Application Strategy Report shows that 41% of companies now manage at least as many APIs as #apps. Even more startling, the largest global companies (with annual revenues of over $10 billion USD) manage an average of nearly 1,400 APIs—with a handful managing as many as 10,000 APIs! As more companies adopt #AI, the growth in APIs is only expected to accelerate. To manage the security challenges of API growth. More than 40% of respondents are automating their app and API security infrastructures. And almost all (95%) are turning to API gateways as a valuable layer of protection. This is just the beginning. With APIs increasingly strategic to the business, API security is imperative—and the best-protected organizations will take an even more comprehensive approach. For more on this and other critical insights, be sure to check out our 10th annual State of Application Strategy Report, which went live today. Also, stay tuned as I share more insights from this report in the coming months: https://go.f5.net/sod5x6xk

178

Learn how to power real-time data streaming applications with Couchbase Capella and Amazon MSK in our latest blog post

3

Unlocking Throughput & Flow, from the book "The Goal" written by Eliyahu Goldratt In this video, I'm diving deep into the goldmine of insights from Eliyahu Goldratt's "The Goal." We'll explore powerful concepts like Throughput (the money your system generates), Bottlenecks (the weak link slowing you down), and the magic of Flow (getting things done efficiently). Discover how to: Measure what truly matters: I'll explain the difference between inventory (money tied up in stuff) and Throughput (money coming in). Identify your bottleneck: Find the slowest link in your chain and optimize for it, not local efficiencies. Master the art of Flow: I'll show you how to reduce batch sizes, minimize wait times, and keep things moving smoothly. Implement powerful tools: I'll delve into Kanban systems and Drum-Buffer-Rope (DBR) to improve flow and prevent overproduction. #eliyahu goldratt, # the goal eliyahu goldratt, # business book summary, # business book review, # productivity tips, # bottleneck theory, # throughput accounting, # kanban system, # drum buffer rope, # business efficiency, # business growth, # throughput, # inventory management, # operational expenses, # dependent events, # statistical fluctuations, # batch size reduction, # lead time reduction, # theory of constraints, entrepreneurship, # small business, # manufacturing, # operations management, # project management, # supply chain management

53

4 Comments

Multi cloud GenAI needs, with all the building blocks for modern applications, from MongoDB...

10

We are SO excited to unveil our partnership with New Relic and our membership of the Secure Developer Alliance. Our industry is still very early in understanding how to secure our developers. Our first step in the alliance will be focused on education — where we’ll be working with Gigamon,��Lacework, Aviatrix, and Opus Security to publish: - Best practices for implementing and managing observable DevSecOps processes and compliance - Guidance on using technologies and tools conducive to the adoption of observable DevSecOps - Requirements, benefits, and approaches to DevSecOps for application security Beyond training, we expect to share more news about technology integrations between New Relic / FOSSA to tie build-time and runtime vulnerability insights together for developers. As Krystle Portocarrero summarizes in our announcement: "Together we’ll offer a seamless, integrated experience that empowers developers to innovate in the cloud and confidently leverage the full spectrum of open-source software to accelerate their projects while maintaining a strong security posture." New Relic is leading the pack on innovating in how observability can distill the signal out of noise coming from cloud security tooling.  I encourage everyone to try New Relic Vulnerability Management and follow the SDA to see what is possible. Huge thanks to our many friends at New Relic that made this possible — Krystle Portocarrero, Oren Ben-Shaul, Manav Khurana, Ashan Willy, Peter Pezaris.  Excited for what’s next!! Read More: https://lnkd.in/eDGEV2yZ

90

12 Comments

#Kafka is so popular... 13.5k forks, thousands of users! ⁉ But how many of these users have exactly once event processing ⁉ 👉 Within DBOS transact, get exactly once event processing with a single annotation! 👉 You don't need 2 years in CS school to get this distributed system problem right anymore :) #Confluent #nodejs #typescript

13

Heard about Conviva time-state analytics but not sure what this novel technology to compute stateful metrics (which we use to measure and improve users QoE in realtime) is? Here are two quick examples that show how to build a stateful metric without writing a complex and very expensive SQL query. Video Streaming use case: https://lnkd.in/dHHpqetq Retail use case: https://lnkd.in/dHHpqetq

41

Fellow in-house taxonomists, ontologists, metadata managers, and semantic wizards: what is the name of the team or business unit that you work within? I have the good fortune of getting to rename the "Content Standards" program I lead at Kaiser Permanente IT. My team has determined that the name is too vague to convey the core activities of the program, and leads to confusion with the cross-functional teams we collaborate with. I'm looking for a clear name that will resonate with stakeholders within my org. So I've got a very classic taxonomist question: What to call the program? 🤔 For some context... What my program is responsible for: - Defining and governing standards for metadata, structured data, taxonomy, and ontology for classifying/structuring digital content - Implementing processes and platforms to good metadata hygiene - Assessment and remediation of content metadata What my program doesn't cover: - Writing or editorial style guides - Content design - Content strategy - Site maps or navigational structure - Data governance Any ideas are most welcome!

36

48 Comments

There comes a point when all software companies need to evolve their #authorization strategy. In many cases, a move to a finer-grained authorization model is the answer. I penned some thoughts here: Need to evolve your authorization strategy? Five questions to ask https://lnkd.in/eriQB2Ri via Okta Auth0 by Okta #developers #SaaS #FGA

63

1 Comment

At Elemental Cognition, we’re transforming #GenAI solutions using #ApacheKafka and Confluent. In this The New Stack article, I shared how our AI platform uses Kafka and Confluent Cloud to stream and vet data in real-time before the EC Reasoning Engine applies formal logic for transparent, accurate solutions. Check out the article for a deep dive into the best practices we’ve learned for building reliable GenAI apps. https://lnkd.in/eU5RupZu

36

4 Comments

Large Language Models (#LLMs) are evolving rapidly, and the demand for running them at scale is skyrocketing. #Ollama is stepping up to meet this need with its official container image on Docker Hub, excelling in local #deployments. However, there’s a catch for containerized environments: downloading #models requires the ollama serve command to be running within the #container. This extra step complicates automated deployments via #CI #CD #pipelines, making seamless model access challenging. To take your #model to production, it needs to be #container-ready—a feature Ollama has yet to fully support. The gap in #AIOps for smooth deployments gets blurred with the inference services available as an easy way to take it to production. Goal is to overcome obstacles and deliver solutions 😉 so here is to get you started on having your (O)#llama #production ready in a private #deployment from your #local to #cloud all in #oneshot . Enjoy Medium: https://lnkd.in/g9jqBQ8d dev.to: https://lnkd.in/gi_gGwam #AI #LLM #Ollama #llama3 #phi3 #mistral #S6overlay #AIOps #CI #CD #oneshot #container #docker #podmanEnjoy

11

What OpenAI's GPT-4o release means for the future of SaaS -- I'll break down the threat in my new article for Inc. Magazine https://lnkd.in/eUvZ4Ywq #innovation #technology #AI #software #saas

3

Generative AI is changing the data storage landscape as file sizes continue to grow. My latest piece for Fast Company Executive Board on why file sizes are growing and what organizations can do to manage them in a cloud environment. https://lnkd.in/gNDT3-Fh #GenerativeAI #Cloud #ContentManagement

160

3 Comments

"Developers love MongoDB and the experience around it" Says Sahir Azam as he gave some amazing insights into MongoDB's future vision and the company's passion for empowering developers with intelligent data tools. It is admirable how our team has strengthened bonds with incredible cloud providers like AWS, Azure, and Google to use the power of large language models and enhance our database ecosystem. By coupling our modern data platform with advanced AI, we are committed to always striving to enrich the inner loop of the developer experience. #MongoDB #artificialintelligence #generativeai

31

When people ask me about Cube of course I explain semantic layers, and talk about the organizational benefits such as consistency, security, governance and developer experience. Those are all great things organizations should be interested in, but really I'm a database nerd at heart. I also look after the alliances function at Cube, including our integrations with all of our wonderful ISV partners so interoperability is always top of mind for me. To look deeper at Cube is to understand that one of our big differentiators is our SQL API and all of the query rewrite and translation work we do that ensures that any front-end data experience works with any of our data source partners. This is a particularly gnarly problem that involves making sure that any odd SQL that your BI tool generates is something that we can parse, understand, optimize, and execute against our pre-aggregations, or any connected data source. This great post from our CTO Pavel Tiunov and Head of Product Igor Lukanin goes into great detail about how the SQL API works and our efforts to improve it. https://lnkd.in/gTSxFRam

14

1 Comment

Sweet use of Streamlit chat UI elements to build a RAG-based document chatbot in Snowflake.

11