Support multiline log entries (stack traces)

[zhiqiang-zz]

Problem

Current implementation splits logs by newline (\n), which breaks multiline log entries like:

• Rust/Java stack traces
• JSON blobs spanning multiple lines
• Error messages with context

Each line is treated as a separate log entry, causing patterns to fail and incorrect categorization.

Proposed Solution

Use a "first-line pattern" approach (same as Filebeat/Logstash/Fluentd):

1. Detect entry start pattern - LLM analyzes sample logs and identifies regex that matches the START of a new log entry (e.g., timestamp + log level)
2. Merge continuation lines - Lines not matching the pattern are appended to the previous entry
3. Process merged entries - Patterns and pipeline work on complete log entries

Implementation

New function: detectFirstLinePattern()

• LLM call to detect the first-line regex
• Returns pattern like ^\d{4}-\d{2}-\d{2}.*?(DEBUG|INFO|WARN|ERROR)

New function: mergeMultilineEntries()

function mergeMultilineEntries(lines: string[], firstLinePattern: RegExp): string[] {
  const entries: string[] = [];
  let currentEntry = "";
  for (const line of lines) {
    if (firstLinePattern.test(line)) {
      if (currentEntry) entries.push(currentEntry);
      currentEntry = line;
    } else {
      currentEntry += "\n" + line;
    }
  }
  if (currentEntry) entries.push(currentEntry);
  return entries;
}

CLI flag

• --first-line-pattern <regex> - skip LLM detection, use provided pattern

References

• Filebeat multiline
• Fluentd multiline parser
• Datadog multiline guide

[STRRL]

Multiline detection breaks on embedded timestamps in JSON payloads

The current Detector.IsNewEntry() only checks whether a line starts with something that looks like a timestamp (scanning up to MaxScanBytes=60 bytes from the beginning). This works for most cases, but breaks when a log entry contains multi-line JSON with embedded timestamp-like values.

Example

2024-03-28T13:45:30.123Z INFO  Received payload:
{
  "event": "deploy",
  "created_at": "2024-03-28T13:44:00.000Z",
  "metadata": {
    "scheduled_at": "2024-03-28T12:00:00.000Z"
  }
}

The detector sees "created_at": "2024-03-28T13:44:00.000Z" and scores the line as a new entry because the timestamp pattern matches within the first 60 bytes. The merged output incorrectly splits this into multiple log entries.

Root cause

IsNewEntry scans the first N bytes of each line for timestamp-like token sequences using the token graph. It has no awareness of whether the line is part of a structured block (JSON, XML, YAML) that belongs to the previous entry.

Possible approaches

1. Indentation / structural heuristic: Lines starting with whitespace, {, }, [, ], or "key": are very likely continuations, not new entries. A cheap pre-check could skip timestamp detection for these lines entirely.

2. Position-aware scoring: Only match timestamps that appear at the very start of the line (column 0), not embedded within a key-value pair. The current MaxScanBytes window does not enforce this — a line like "created_at": "2024-03-28..." still gets scanned.

3. Bracket-depth tracking in Merger: Track open {/[ counts across lines. While inside an unclosed block, suppress new-entry detection. This handles nested JSON but adds state to the merger.

4. Hybrid: Combine (1) and (2) — require timestamps to be at column 0 (or preceded only by a log level / hostname prefix), AND skip lines that look structurally like continuations.

Approach (1) is the cheapest and handles the most common case. Approach (3) is the most robust but changes the merger's contract.

[STRRL]

Research: How the industry handles this

Only Datadog solves it

Surveyed Datadog Agent, Fluent Bit, Vector, Promtail, and Logstash. Only Datadog has automatic multiline detection with JSON-aware preprocessing. Everyone else requires user-configured regex and punts the problem to the user.

Datadog's solution is a two-layer design:

1. JSONAggregator — runs before timestamp detection, uses an incremental JSON validator to track {}/[] depth and merge pretty-printed JSON into single lines
2. JSONDetector heuristic — ^\s*\{\s*["}] check runs before TimestampDetector in the labeler chain, short-circuits JSON lines to skip timestamp matching entirely

Vector has the same problem (unresolved)

Vector (Rust, MPL-2.0) has multiple open issues about multiline JSON:

• #4952 — "Better support multiline JSON event ingestion" (open since 2020)
• #20932 — CrowdStrike multiline JSON not working (2024)
• #1559 — closed, told users to write Lua transforms
• #6486 — misbehaving multiline behavior
• #3464 — huge multi-line logs handled incorrectly

Considered approaches

Option A: Embed Vector — Use Vector as a library/subprocess for log parsing.

• ❌ Rust codebase (133MB), LAPP is Go — means CGO or subprocess overhead
• ❌ Vector itself hasn't solved multiline JSON (see issues above)
• ❌ MPL-2.0 vs LAPP's MIT
• ❌ Vector is a data pipeline (collect→transform→ship), not a pattern discovery tool — architectural mismatch

Option B: Port Datadog's JSON preprocessing layer — Add JSONAggregator + IncrementalJSONValidator + JSONDetector to LAPP's existing multiline pipeline.

• ✅ LAPP already uses Datadog's token graph and timestamp detector — same architecture
• ✅ Same language (Go), straightforward port
• ✅ Apache-2.0 compatible with MIT
• ✅ ~200-300 lines of Go to add
• ✅ Actually solves the problem (proven in production at Datadog scale)

Recommendation

Go with Option B. The missing piece is small and well-scoped.

[STRRL]

Closed: Multiline log support merged in PR #5.