Sebas-AD-https

config/slips.yaml

@@ -215,6 +215,51 @@ flowmldetection:
   # 'Malicious' data in order for the test to work.
   mode: test
 
+#############################
+anomaly_detection_https:
+  # Number of initial hours used to train the baseline model assuming benign traffic.
+  # If set to 0, detection starts immediately and baseline is learned online.
+  training_hours: 0
+
+  # Thresholds for anomaly detection in hourly and per-flow checks.
+  hourly_zscore_threshold: 3.0
+  flow_zscore_threshold: 3.5
+
+  # If hourly anomaly score is lower than this, treat it as drift and adapt.
+  adaptation_score_threshold: 2.0
+
+  # Adaptation speed for baseline updates.
+  baseline_alpha: 0.1
+  drift_alpha: 0.05
+  suspicious_alpha: 0.005
+
+  # Minimum baseline history required before z-score checks are applied.
+  min_baseline_points: 6
+
+  # Max flow anomalies in an hour still considered "small drift".
+  max_small_flow_anomalies: 1
+
+  # Optional ADWIN-based drift trigger (river).
+  # If true, drift/suspicious update is decided only after ADWIN signals drift.
+  # If false, Slips uses the previous threshold-only drift logic.
+  use_adwin_drift: true
+  adwin_delta: 0.002
+  adwin_clock: 32
+  adwin_grace_period: 10
+  adwin_min_window_length: 5
+
+  # JA3 statistical fallback gate used only when training_hours = 0.
+  # If there is no benign training period, ja3_changes is not scored
+  # until this minimum hourly value is reached.
+  ja3_min_variants_per_server: 3
+
+  # Operational logging verbosity:
+  # 0: disabled
+  # 1: important events only (detections, drift, training fit, stop/start)
+  # 2: + hourly summaries (recommended)
+  # 3: + per-flow arrivals and model update details
+  log_verbosity: 3
+
 #############################
 virustotal:
   # This is the path to the API key. The file should contain the key at the

docs/detection_modules.md

@@ -126,6 +126,12 @@ tr:nth-child(even) {
 
 </table>
 
+## HTTPS Anomaly Detection Module
+
+For the full technical description of the HTTPS anomaly detector (features, training, adaptation, z-score logic, evidence format, and configuration), see:
+
+- [HTTPS Anomaly Detection](https_anomaly_detection.md)
+
 
 
 ### Virustotal Module

docs/https_anomaly_detection.md

@@ -0,0 +1,192 @@
+# HTTPS Anomaly Detection Module
+
+This document describes how the `anomaly_detection_https` module detects anomalies from TLS/HTTPS traffic in Slips.
+
+![HTTPS anomaly detection example](images/Slips-AD-example.png)
+
+## Goal
+
+Detect unusual HTTPS behavior per host, using:
+
+- Hourly behavior changes (volume and novelty patterns).
+- Flow-level deviations (for known servers).
+- Adaptive baselines that update over time, with poisoning resistance.
+
+## Input data used
+
+The module subscribes to SSL/TLS events and reads related connection metadata from DB for the same UID.
+
+Main fields used:
+
+- SSL: `uid`, `server_name` (SNI), `ja3`, `ja3s`, `dport`, `sport`
+- Conn (correlated): destination IP, total bytes, timing info
+
+## Traffic-time logic
+
+All detection windows are based on **traffic timestamps** (packet/log time), not wall clock time.
+
+This keeps behavior consistent for:
+
+- live interface capture,
+- live Zeek folder input,
+- offline PCAP,
+- offline Zeek logs.
+
+## Features
+
+The module computes per-host hourly features:
+
+- `ssl_flows`: number of SSL flows in the hour.
+- `unique_servers`: number of distinct destination servers.
+- `new_servers`: number of servers not seen before for that host.
+- `ja3_changes`: number of new JA3 variants seen per server in the hour.
+- `known_server_avg_bytes`: mean bytes for flows to already-known servers.
+
+Flow-level feature:
+
+- `bytes_to_known_server`: per-server bytes deviation on each flow.
+
+## Baseline and training
+
+Each host has independent models.
+
+### Training phase (`training_hours > 0`)
+
+For the first configured benign hours, the module does **fit-only** (Welford online moments):
+
+- no detection decisions are emitted from hourly z-score rules before training ends,
+- baseline mean/variance are learned strongly from this period.
+
+### No explicit training (`training_hours = 0`)
+
+Detection starts immediately using online adaptation.
+
+Special fallback only for `ja3_changes`:
+
+- if hourly `ja3_changes < ja3_min_variants_per_server`, that hourly signal is ignored until enough activity exists.
+
+## Scoring
+
+Each modeled feature uses z-score:
+
+- `z = |x - mean| / std_effective`
+- `std_effective` uses variance with a robust minimum floor to avoid unstable near-zero std.
+
+Thresholds:
+
+- `hourly_zscore_threshold` for hourly features
+- `flow_zscore_threshold` for flow bytes to known servers
+
+## Adaptation states
+
+After each hour closes, the module chooses model update mode:
+
+1. `training_fit`  
+   During benign training: Welford fit (no EWMA alpha).
+
+2. `drift_update`  
+   If anomaly score is small (`hourly_score <= adaptation_score_threshold`) and flow anomaly count is small (`<= max_small_flow_anomalies`), update with `drift_alpha`.
+
+3. `suspicious_update`  
+   Otherwise update with `suspicious_alpha` (much smaller), to limit poisoning.
+
+For normal non-anomalous periods outside training, per-feature EWMA uses `baseline_alpha`.
+
+### Optional ADWIN drift trigger
+
+If `use_adwin_drift=true` and `river` is installed, ADWIN is used as drift trigger in both paths:
+
+- **Hourly path**: ADWIN receives `hourly_adwin_score` (sum of hourly feature z-scores).
+- **Flow path**: ADWIN receives `flow_score` (sum of reason z-scores, novelty reasons mapped to a small fixed score).
+- ADWIN drift detected -> classify as `drift_update` or `suspicious_update` using existing thresholds.
+- No ADWIN drift -> use `baseline_update` (`baseline_alpha`).
+- During benign training, ADWIN is still warmed with benign scores to reduce cold-start noise after training.
+
+Why raw signals:
+
+- drift is a distribution change in the observed variables, so ADWIN tracks the raw feature streams directly,
+- z-scores are still used for anomaly magnitude and evidence reasons, but not as the primary drift input.
+
+Performance note:
+
+- hourly ADWIN cost scales with hourly feature count,
+- flow ADWIN cost scales with per-flow signal count,
+- both are constant-time scalar updates and usually lightweight.
+
+## New server vs JA3 behavior
+
+- `new_servers` is modeled as an hourly statistical feature and adapted over time.
+- `new_server` can also appear as a direct flow-level novelty reason.
+- `ja3_changes` is handled statistically at hourly level (with fallback gate only when training is zero).
+- `new_ja3s` can appear as direct flow-level novelty reason.
+
+## Confidence and threat level
+
+Each detection computes confidence score `[0,1]` from multiple factors:
+
+- anomaly severity,
+- persistence in recent history,
+- baseline quality,
+- multi-signal agreement.
+
+Mapped levels:
+
+- low / medium / high confidence
+
+Threat level used in evidence:
+
+- `low` for low or medium confidence
+- `medium` for high confidence
+
+## Evidence format
+
+Evidence description is human-readable and concise:
+
+`HTTPS anomaly: type=<type>; confidence=<level> (<score>); reason=<reason>; value=<value>; why=<explanation>.`
+
+Examples of reasons:
+
+- New Server
+- New JA3S
+- Bytes to Known Server
+- Hourly feature deviations (e.g., New Servers Count, JA3 Changes)
+
+## Configuration keys
+
+Section: `anomaly_detection_https` in `config/slips.yaml`.
+
+Main keys:
+
+- `training_hours`
+- `hourly_zscore_threshold`
+- `flow_zscore_threshold`
+- `adaptation_score_threshold`
+- `baseline_alpha`
+- `drift_alpha`
+- `suspicious_alpha`
+- `min_baseline_points`
+- `max_small_flow_anomalies`
+- `ja3_min_variants_per_server`
+- `use_adwin_drift`
+- `adwin_delta`
+- `adwin_clock`
+- `adwin_grace_period`
+- `adwin_min_window_length`
+- `log_verbosity`
+
+Default: `use_adwin_drift=true`.
+
+Reference:
+
+- River ADWIN: https://riverml.xyz/latest/api/drift/ADWIN/
+
+## Operational logs
+
+The module logs key events such as:
+
+- flow arrivals,
+- hour close and computed features,
+- training fit updates,
+- drift updates,
+- suspicious updates,
+- detections and emitted evidence.

docs/index.rst

@@ -17,6 +17,8 @@ This documentation gives an overview how Slips works, how to use it and how to h
 
 - **Detection modules**. Explanation of detection modules in Slips, types of input and output. See :doc:`Detection modules <detection_modules>`.
 
+- **HTTPS anomaly detection**. Detailed design and behavior of the HTTPS anomaly detector. See :doc:`HTTPS anomaly detection <https_anomaly_detection>`.
+
 - **Architecture**. Internal architecture of Slips (profiles, timewindows), the use of Zeek and connection to Redis. See :doc:`Architecture <architecture>`.
 
 - **Training with your own data**. Explanation on how to re-train the machine learning system of Slips with your own traffic (normal or malicious).See :doc:`Training <training>`.
@@ -49,6 +51,7 @@ This documentation gives an overview how Slips works, how to use it and how to h
    usage
    architecture
    detection_modules
+   https_anomaly_detection
    flowalerts
    features
    training

managers/process_manager.py

@@ -22,6 +22,8 @@
 from typing import (
     List,
     Tuple,
+    Dict,
+    Iterable,
 )
 
 from exclusiveprocess import (
@@ -52,6 +54,8 @@
 class ProcessManager:
     def __init__(self, main):
         self.main = main
+        # Can be used by signal handlers before startup finishes.
+        self.processes: List[Process] = []
 
         # this is the queue that will be used by the input proces
         # to pass flows to the profiler