A comprehensive SIEM and endpoint detection engineering project - catch what was not supposed to be seen.
GhostTrace is a hands-on detection engineering project that builds a full endpoint telemetry pipeline using Elastic Cloud, Elastic Defend, and Fleet, with a Kali Linux VM as the test endpoint. The lab walks through agent enrollment, adversary-style activity generation, log hunting, building a SIEM dashboard from scratch, and writing custom detection rules.
The goal is to simulate the complete workflow a SOC analyst or detection engineer would practice in a controlled environment: generate activity, collect telemetry, hunt the evidence, visualize the pattern through a custom dashboard, and convert repeatable behavior into alerts.
- Deploy an Elastic Cloud environment and access Kibana
- Enroll a Kali Linux VM into Elastic Fleet with Elastic Defend
- Generate endpoint activity with Nmap, gobuster, hydra, and baseline Linux commands
- Query telemetry in Kibana Logs and Security
- Build dashboard visualizations for endpoint event volume
- Write custom detection rules with severity mapping
- Validate alerts in Security > Alerts
Kali Linux VM
`-- Elastic Agent (Fleet-managed, Elastic Defend policy)
`-- Elastic Cloud Deployment
|-- Kibana Logs -> telemetry search and hunting
|-- Security Alerts -> rule-triggered detections
`-- Dashboards -> event volume and severity views
| Component | Role |
|---|---|
| Elastic Cloud | Hosts the Elastic deployment and Kibana interface |
| Elastic Defend | Collects endpoint telemetry and security events |
| Fleet | Manages agent enrollment and policy assignment |
| Kali Linux | Lab endpoint for command-line and scan activity generation |
| Nmap | Port scan and service discovery telemetry |
| gobuster / hydra | Higher-risk offensive tooling for alert testing |
| Kibana Dashboards | Event count and severity distribution visualization |
.
|-- README.md
|-- assets/
| `-- screenshots/
|-- docs/
| |-- detection-rules.md
| |-- lab-runbook.md
| `-- troubleshooting.md
`-- .gitignore
1. Spin up the environment
- Create an Elastic Cloud deployment
- Open Kibana > Integrations > install Elastic Defend
- Create a Fleet policy for the lab endpoint
- Copy the Linux enrollment command from Fleet and run it on the Kali VM
2. Confirm agent health
sudo systemctl status elastic-agent.service3. Generate test activity
# Reconnaissance
sudo nmap 127.0.0.1
sudo nmap -sS <target-ip>
sudo nmap -sT <target-ip>
sudo nmap -p- <target-ip>
# Web enumeration
gobuster dir -u http://<target-ip> -w /usr/share/wordlists/dirb/common.txt
# Credential attack simulation
hydra -l test -p test ssh://<target-ip>
# Baseline process noise
ls -l
mkdir ghosttrace-test
ping -c 4 <target-ip>Only scan systems you own or are explicitly authorized to test.
Run these KQL queries in Kibana to confirm telemetry is arriving.
# Nmap activity
process.name : "nmap" or process.args : "nmap"
# Offensive tooling
process.name : ("gobuster" or "hydra") or process.args : ("gobuster" or "hydra")
# Baseline commands
process.name : ("ls" or "mkdir" or "ping") or process.args : ("ls" or "mkdir" or "ping")
# Privileged execution
process.name : "sudo" or process.args : "sudo"
# Normalized scan events, if applicable
event.action : "nmap_scan"| Rule | Query Focus | Severity | Purpose |
|---|---|---|---|
| Nmap Scan Detection | Nmap process and scan flags | High | Detect reconnaissance and port enumeration |
| Offensive Tooling Alert | gobuster and hydra process activity | Critical | Detect higher-risk offensive tooling |
| Baseline Command Monitor | ls, mkdir, ping | Low | Verify endpoint telemetry collection |
| Sudo Usage Review | sudo process and argument activity | Medium | Surface privileged command execution |
Full rule logic: docs/detection-rules.md
Built in Kibana > Analytics > Dashboards using Lens visualizations:
- Area chart - event count over time (
@timestamp) - Bar chart - process volume by command family
- Severity distribution - low / medium / high / critical alert spread
- Data table - host, user, process, command line, event action
The dashboard answers three questions at a glance:
- Which endpoint generated activity?
- Which command family caused the spike?
- Which alerts need triage first?
1. Generate controlled activity from Kali
2. Confirm Elastic Agent health in Fleet
3. Search Logs for process telemetry
4. Pivot into event details and inspect process.name, process.args, host.name, user.name, @timestamp
5. Build dashboard to compare event families
6. Convert repeatable patterns into custom query rules
7. Validate alerts in Security > Alerts
8. Tune severities and suppress noisy baseline commands
This section documents the lab build in the same order the environment was deployed, tested, and validated.
Start by creating an Elastic Cloud account and launching a default deployment. This deployment hosts Kibana, Fleet, Elastic Defend, dashboards, and alerting.
After the deployment is created, confirm that the Elastic Cloud console shows the environment as available.
From Kibana, open the main navigation menu and go to Integrations. This is where the endpoint security integration is installed.
Search for Elastic Defend and open the integration.
Configure the Elastic Defend integration and attach it to a Fleet-managed agent policy. The policy controls what the enrolled Kali endpoint collects and forwards.
Add the integration to the selected policy.
Open the add-host workflow in Fleet so the Kali endpoint can be enrolled.
Copy the Linux enrollment command generated by Elastic.
Run the enrollment command on the Kali Linux VM. Once complete, Elastic Agent starts sending endpoint telemetry to the Elastic deployment.
Validate the agent service with systemd.
sudo systemctl status elastic-agent.service
Generate controlled reconnaissance activity with Nmap. These commands create process telemetry and network-scan evidence that can be hunted in Kibana.
sudo nmap 127.0.0.1
sudo nmap -sS <target-ip>
sudo nmap -sT <target-ip>
sudo nmap -p- <target-ip>
Run additional scans against authorized lab targets to create more event variety.
Open Observability > Logs to search the telemetry forwarded by Elastic Agent.
Search for Nmap-related process activity.
process.name : "nmap" or process.args : "nmap"
Open individual log entries to inspect event fields and confirm the command-line evidence.
The event details show process arguments and other fields needed for detection rule logic.
Expand the event record to review the full context for the generated command.
Go to Analytics > Dashboards and create a dashboard for endpoint event visualization.
Create a Lens visualization and choose a chart type such as area or bar.
Start with an empty Lens editor and build the visualization manually.
Use @timestamp for the horizontal axis so event volume can be viewed over time.
Use count as the vertical metric to measure the number of events.
Save the visualization once the event count appears correctly.
Add multiple views to compare event volume and alert distribution in one dashboard.
Open Security > Alerts, then go to rule management.
Create a new rule from the rule management screen.
Select a custom query rule so the alert logic can be based on the telemetry observed during hunting.
Enable the rule after setting the query, severity, schedule, and actions.
Create a high-severity rule for Nmap scan activity.
process.name : "nmap" or process.args : "nmap" or process.args : ("-sS" or "-sT" or "-p-")
Create a critical-severity rule for offensive tooling such as gobuster and hydra.
process.name : ("gobuster" or "hydra") or process.args : ("gobuster" or "hydra")
Create a low-severity rule for baseline Linux commands to confirm endpoint process collection.
process.name : ("ls" or "mkdir" or "ping") or process.args : ("ls" or "mkdir" or "ping")
Run another controlled command sequence from Kali to trigger the rules.
gobuster dir -u http://<target-ip> -w /usr/share/wordlists/dirb/common.txt
hydra -l test -p test ssh://<target-ip>
ls -l
ping -c 4 <target-ip>
sudo nmap -sS <target-ip>
sudo nmap -p- <target-ip>
Generate more Nmap activity to confirm the scan rule fires consistently.
Review Security > Alerts to confirm the expected low, high, and critical severity alerts appear.
- Add Windows endpoint telemetry with Sysmon and Elastic Agent
- Add Linux authentication failure rules for SSH and sudo misuse
- Map all rules to MITRE ATT&CK: Discovery, Credential Access, Execution
- Add response actions: Slack, email, or webhook notifications
- Export detection rules as saved objects or NDJSON
- Build a triage worksheet template for each alert type
GhostTrace demonstrates a complete detection engineering loop: endpoint enrollment, telemetry generation, SIEM ingestion, log hunting, visualization, and alert engineering. Small enough to run as a personal lab. Structured like a production SOC workflow.
Built by Austin BC