Totally agree with at Omer Singer on the value of running your threat detection on top of a modern cloud database like Snowflake. And if you're moving your SOC to Snowflake, you should look at Outpost Ventures portfolio company Anvilogic, a SOC and threat detection platform purpose-built for cloud databases like Snowflake!
What’s your Goldilocks latency for threat detection? Some security architects prioritize speed: real-time measured in seconds. On the other side of the spectrum, I know of at least one popular security product that issues alerts once an hour. Consider these factors and share your latency preferences in the comments below. Too Fast: The infosec instinct says “I want detections as near real time as possible.” But that comes with a heavy price. Not necessarily in terms of infrastructure spend, though streaming analytics can be costly. The bigger issue with looking at live events “on the wire” is the impact to detection fidelity. The uncomfortable truth about real-time detections is that they rely too much on the content of the individual event. It’s like if a criminal investigation could only involve what the suspect carried during the arrest. In cyber terms, good context covers the user’s behavioral profile, the environment where the action took place and an up to date model of threat actor techniques. None of which can be captured well in a static rule. In response to these limitations, some organizations rely on streaming analytics platforms like Spark or Flink combined with low latency data platforms like Redis or DynamoDB. This introduces complexity, overhead and infrastructure cost while still facing significant limitations on context and flexibility. There’s also an impact on who from the team can participate in the detection engineering. Too Slow: Applying batch analytics to threat detection opens up all the history, details and models that weren’t available in the stream. But in certain situations a late detection is almost as bad as no detection. For example, guardrails around production code deployment aren’t going to be effective if they trigger an hour after vulnerable code shipped or sensitive data leaked. Just Right: If single-second detection latency is too noisy and single-hour latency is too laggy, Goldilocks might be in the single minutes. Now is a good time to start thinking about this balance. Snowflake is blurring the lines between batch and stream analytics, and while sub-second end to end latency is still out scope, we’re reaching a point where lag in the single minutes is doable. This evolution could change the math for where many organizations choose to do their detections. #securitydatalake
