Large number of "[k8saudit-aks] cannot parse JSON: cannot parse number" errors

[bc-averlon]

Describe the bug

I've deployed Falco with the k8saudit-aks plugin and configured it to read from an eventhub. Events are being processed but there is a lot of log output for failed parsing. After running Falco for ~20 minutes there were over 9100 of these errors output from a single pod.

2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475483 1 clustersta...or nodeGroup aks-agentpool-20679993-vmss"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475573 1 static_autoscaler.go:548] No unschedulable pods"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475590 1 azure_scal...679993-vmss, returning in-memory size: 2"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475653 1 pre_filter...up min size reached (current: 2, min: 2)"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475708 1 pre_filter...up min size reached (current: 2, min: 2)"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475944 1 azure_scal...679993-vmss, returning in-memory size: 2"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475964 1 azure_scal...679993-vmss, returning in-memory size: 2"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:24:54.475980 1 orchestrat...pToNodeGroupMinSize: scale up not needed"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "I"; unparsed tail: "I0109 21:25:17.623680 1 trace.go:2...2026 21:25:16.342) (total time: 1281ms):"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "T"; unparsed tail: "Trace[7816759]: ---"limitedReadBody succeeded" len:629048 11ms (21:25:16.354)"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "T"; unparsed tail: "Trace[7816759]: ---"Conversion done" 52ms (21:25:16.406)"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "T"; unparsed tail: "Trace[7816759]: ["GuaranteedUpdate etcd3...iextensions.k8s.io 1212ms (21:25:16.411)"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "T"; unparsed tail: "Trace[7816759]: ---"About to Encode" 1027ms (21:25:17.462)]"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "T"; unparsed tail: "Trace[7816759]: ---"Write to database ca...cceeded" len:629048 123ms (21:25:17.586)"
2026/01/09 21:35:56 [k8saudit-aks] cannot parse JSON: cannot parse number: unexpected char: "T"; unparsed tail: "Trace[7816759]: [1.281198436s] [1.281198436s] END"

I looked at other issues and found a similar one for EKS but as far as I can tell that's a different issue.
#297

I couldn't find specific documentation on which diagnostic settings should be sent to the event hub but I realized as I was writing this up that could be the issue if only kube-audit logs are parsed by the plugin.

I'm currently sending all of these:

kube-apiserver
kube-controller-manager
kube-scheduler
cluster-autoscaler
kube-audit

How to reproduce it

• Enable diagnostic settings for the target AKS cluster:

locals {
  storage_categories = [
    "kube-apiserver",
    "kube-controller-manager",
    "kube-scheduler",
    "kube-audit"
  ]
}

# Storage Account Diagnostic Settings
resource "azurerm_monitor_diagnostic_setting" "storage" {
  name               = "${var.cluster_name}-logs-storage"
  target_resource_id = data.azurerm_kubernetes_cluster.aks.id
  storage_account_id = azurerm_storage_account.logs.id

  dynamic "enabled_log" {
    for_each = toset(local.storage_categories)
    content {
      category = enabled_log.value
    }
  }
}

• Deploy Falco via the official Helm chart:

  falco:
    rules_files:
      - /etc/falco/k8s_audit_rules.yaml
      - /etc/falco/rules.d
    plugins:
      - name: k8saudit-aks
        library_path: libk8saudit-aks.so
        init_config:
          blob_storage_container_name: "<blob-storage-container-name>"
          blob_storage_connection_string: "<blob-storage-connection-string>"
          event_hub_name: "<event-hub-name>"
          event_hub_namespace_connection_string: "<event-hub-ns-connection-string>"
        open_params: "http://:9765/k8s-audit"
      - name: json
        library_path: libjson.so
        init_config: ""
    load_plugins: [k8saudit-aks, json]

Expected behaviour

Expected behavior is for the events to parse properly without logging errors.

Environment

• Falco version: 0.42.1
• System info: Linux version 5.15.0-1101-azure (buildd@lcy02-amd64-088) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) refactor(plugins/k8saudit): make evt size and batch size configurable with official terminology #110-Ubuntu SMP Wed Nov 19 02:16:35 UTC 2025
• Cloud provider or hardware configuration: Azure AKS 1.33.5
• OS: AKSUbuntu-2204gen2containerd-202512.06.0
• Installation method: Helm

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale