I think the real problem with LLMs is we have deterministic expectations of non-deterministic tools. We’ve been trained to expect that the computer is correct.

Personally, I think the summaries of alerts is incredibly useful. But my expectation of accuracy for a 20 word summary of multiple 20-30 word summaries is tempered by the reality that’s there’s gonna be issues given the lack of context. The point of the summary is to help me determine if I should read the alerts.

LLMs break down when we try to make them independent agents instead of advanced power tools. Alot of people enjoy navel gazing and hand waving about ethics, “safety” and bias… then proceed to do things with obvious issues in those areas.

Larger LLMs can summarize all of this quite well though.

Determinism isn't the issue though. Many responses are fine. The displayed one is bad, whether chosen deterministically or not. Some alternatives:

- Passed out drunk

- Crashed in bed

- Slacking because drunk

...

The issue isn't a lack of context; it's that even the available context was handled poorly.