Fix/BED-7996 improve first time setup #2
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Example configuration: https://bloodhound.specterops.io/openhound/configuration | ||
| [runtime] | ||
| http_show_error_body = true | ||
| log_cli_level = "WARNING" | ||
| log_format = "JSON" | ||
| log_rotate_when = "midnight" | ||
|
|
||
| [extract] | ||
| workers = 8 | ||
|
|
||
| [normalize] | ||
| workers = 3 | ||
|
|
||
| [load] | ||
| delete_completed_jobs = true | ||
| truncate_staging_dataset = true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Example configuration for github secrets: https://bloodhound.specterops.io/openhound/collectors/github/collect-data#example-configuration | ||
| [sources.source.github.credentials] | ||
| app_id = "myappid" | ||
| client_id = "myclientid" | ||
| key_path = "/app/.dlt/github.pem" | ||
| org_name = "myorgname" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Example configuration for jamf secrets: https://bloodhound.specterops.io/openhound/collectors/jamf/collect-data#example-configuration | ||
| [sources.source.jamf] | ||
| username = "myusername" | ||
| host = "https://tenant.jamfcloud.com" | ||
| password = "mypassword" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Example configuration for okta secrets: https://bloodhound.specterops.io/openhound/collectors/okta/collect-data#example-configuration | ||
| [sources.source.okta.credentials] | ||
| base_url = "https://mytenant.oktapreview.com" | ||
| client_id = "myclientid" | ||
| private_key_path = "/app/.dlt/okta.json" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| ## Quick-start | ||
|
|
||
| 1. Copy `.dlt-example` to `${HOME}/.dlt`and `docker-compose.yml` to `${HOME}`. | ||
| 2. Fill in your credentials in the toml files. | ||
| 3. Place any required key files (github.pem, okta.json) in `${HOME}/.dlt`. | ||
| 4. Pull image from SpecterOps Docker Hub: `docker pull specterops/openhound:latest` | ||
| or run to pull from docker-compose.yml: `docker compose pull` | ||
| 5. Run all collectors: `docker compose up -d` | ||
| or run a single one: `docker compose up -d collect-jamf preprocess-jamf convert-jamf` | ||
|
|
||
| Example docker-compose file for running OpenHound with Jamf, GitHub, and Okta collectors. | ||
| Collector output is written to local bind-mount directories under `./output/<collector>/`. | ||
|
|
||
| ## WARNING: | ||
| All config and secret files referenced below MUST exist before running | ||
| `docker compose up`. If they are missing, Docker will create them as directories, | ||
| which will cause the collector to fail. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,162 @@ | ||
| x-cli: &cli | ||
| image: specterops/openhound:latest | ||
| restart: no | ||
|
|
||
| # These environment variables can also be set inside the ~/.dlt/config.toml file | ||
| # and act as an example. Configuring openhound can be done using environment variables, | ||
| # configuration files or a combination of both. | ||
| environment: &env | ||
| LOG_LEVEL: INFO | ||
| EXTRACT__WORKERS: 8 | ||
| LOG_CONTAINER: true | ||
|
|
||
| x-jamf: &jamf | ||
| <<: *cli | ||
| volumes: | ||
| - ${HOME}/.dlt/config.toml:/app/.dlt/config.toml:ro | ||
| - ./output/jamf:/tmp/output | ||
| secrets: | ||
| - source: secrets_jamf | ||
| target: /app/.dlt/secrets.toml | ||
|
|
||
| x-github: &github | ||
| <<: *cli | ||
| volumes: | ||
| - ${HOME}/.dlt/config.toml:/app/.dlt/config.toml:ro | ||
| - ./output/github:/tmp/output | ||
| secrets: | ||
| - source: secrets_github | ||
| target: /app/.dlt/secrets.toml | ||
| - source: key_github | ||
| target: /app/.dlt/github.pem | ||
|
|
||
| x-okta: &okta | ||
| <<: *cli | ||
| volumes: | ||
| - ${HOME}/.dlt/config.toml:/app/.dlt/config.toml:ro | ||
| - ./output/okta:/tmp/output | ||
| secrets: | ||
| - source: secrets_okta | ||
| target: /app/.dlt/secrets.toml | ||
| - source: key_okta | ||
| target: /app/.dlt/okta.json | ||
|
|
||
| services: | ||
| # --- Jamf --- | ||
| collect-jamf: | ||
| <<: *jamf | ||
| command: | ||
| - collect | ||
| - jamf | ||
| - /tmp/output/raw | ||
| - --progress=log | ||
|
|
||
| preprocess-jamf: | ||
| <<: *jamf | ||
| command: | ||
| - preprocess | ||
| - jamf | ||
| - /tmp/output/raw/jamf | ||
| - /tmp/output/lookup.duckdb | ||
| - --progress=log | ||
| depends_on: | ||
| collect-jamf: | ||
| condition: service_completed_successfully | ||
|
|
||
| convert-jamf: | ||
| <<: *jamf | ||
| command: | ||
| - convert | ||
| - jamf | ||
| - /tmp/output/raw/jamf | ||
| - /tmp/output/graph | ||
| - --lookup-file=/tmp/output/lookup.duckdb | ||
| - --progress=log | ||
| depends_on: | ||
| preprocess-jamf: | ||
| condition: service_completed_successfully | ||
|
|
||
| # --- GitHub --- | ||
| collect-github: | ||
| <<: *github | ||
| command: | ||
| - collect | ||
| - github | ||
| - /tmp/output/raw | ||
| - --progress=log | ||
|
|
||
| preprocess-github: | ||
| <<: *github | ||
| command: | ||
| - preprocess | ||
| - github | ||
| - /tmp/output/raw/github | ||
| - /tmp/output/lookup.duckdb | ||
| - --progress=log | ||
| depends_on: | ||
| collect-github: | ||
| condition: service_completed_successfully | ||
|
|
||
| convert-github: | ||
| <<: *github | ||
| command: | ||
| - convert | ||
| - github | ||
| - /tmp/output/raw/github | ||
| - /tmp/output/graph | ||
| - --lookup-file=/tmp/output/lookup.duckdb | ||
| - --progress=log | ||
| depends_on: | ||
| preprocess-github: | ||
| condition: service_completed_successfully | ||
|
|
||
| # --- Okta --- | ||
| collect-okta: | ||
| <<: *okta | ||
| command: | ||
| - collect | ||
| - okta | ||
| - /tmp/output/raw | ||
| - --progress=log | ||
|
|
||
| preprocess-okta: | ||
| <<: *okta | ||
| command: | ||
| - preprocess | ||
| - okta | ||
| - /tmp/output/raw/okta | ||
| - /tmp/output/lookup.duckdb | ||
| - --progress=log | ||
| depends_on: | ||
| collect-okta: | ||
| condition: service_completed_successfully | ||
|
|
||
| convert-okta: | ||
| <<: *okta | ||
| command: | ||
| - convert | ||
| - okta | ||
| - /tmp/output/raw/okta | ||
| - /tmp/output/graph | ||
| - --lookup-file=/tmp/output/lookup.duckdb | ||
| - --progress=log | ||
| depends_on: | ||
| preprocess-okta: | ||
| condition: service_completed_successfully | ||
|
|
||
| secrets: | ||
| # Jamf: username + password auth | ||
| secrets_jamf: | ||
| file: ${HOME}/.dlt/secrets_jamf.toml | ||
|
|
||
| # GitHub: GitHub App auth (app_id, client_id, org_name) + private key PEM (github.pem) | ||
| secrets_github: | ||
| file: ${HOME}/.dlt/secrets_github.toml | ||
| key_github: | ||
| file: ${HOME}/.dlt/github.pem | ||
|
|
||
| # Okta: OAuth client credentials (base_url, client_id) + private key JSON (okta.json) | ||
| secrets_okta: | ||
| file: ${HOME}/.dlt/secrets_okta.toml | ||
| key_okta: | ||
| file: ${HOME}/.dlt/okta.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Example configuration: https://bloodhound.specterops.io/openhound/enterprise#full-configuration-example | ||
| [runtime] | ||
| http_show_error_body = true | ||
| log_cli_level = "WARNING" | ||
| log_format = "JSON" | ||
| log_rotate_when = "midnight" | ||
|
|
||
| [extract] | ||
| workers = 8 | ||
|
|
||
| [normalize] | ||
| workers = 3 | ||
|
|
||
| [load] | ||
| delete_completed_jobs = true | ||
| truncate_staging_dataset = true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| [destination.bloodhoundenterprise] | ||
| interval = "300" | ||
| token_id = "client_token_id" | ||
| token_key = "client_token_key" | ||
| url = "bhe_url" | ||
|
|
||
| # Example configuration for github secrets: https://bloodhound.specterops.io/openhound/collectors/github/collect-data#example-configuration | ||
| [sources.source.github.credentials] | ||
| app_id = "myappid" | ||
| client_id = "myclientid" | ||
| key_path = "/app/.dlt/github.pem" | ||
| org_name = "myorgname" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| [destination.bloodhoundenterprise] | ||
| interval = "300" | ||
| token_id = "client_token_id" | ||
| token_key = "client_token_key" | ||
| url = "bhe_url" | ||
|
|
||
| # Example configuration for jamf secrets: https://bloodhound.specterops.io/openhound/collectors/jamf/collect-data#example-configuration | ||
| [sources.source.jamf] | ||
| username = "myusername" | ||
| host = "https://mytenant.jamfcloud.com" | ||
| password = "mypassword" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| [destination.bloodhoundenterprise] | ||
| interval = "300" | ||
| token_id = "client_token_id" | ||
| token_key = "client_token_key" | ||
| url = "bhe_url" | ||
|
|
||
| # Example configuration for okta secrets: https://bloodhound.specterops.io/openhound/collectors/okta/collect-data#example-configuration | ||
| [sources.source.okta.credentials] | ||
| base_url = "https://mytenant.oktapreview.com" | ||
| client_id = "myclientid" | ||
| private_key_path = "/app/.dlt/okta.json" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| ## Quick-start | ||
|
|
||
| 1. Copy `.dlt-example` to `${HOME}/.dlt`and `docker-compose.yml` to `${HOME}`. | ||
| 2. Fill in your credentials in the toml files. | ||
| 3. Place any required key files (github.pem, okta.json) in `${HOME}/.dlt`. | ||
| 4. Pull image from SpecterOps Docker Hub: `docker pull specterops/openhound:0.1.1-enterprise` | ||
|
Member
There was a problem hiding this comment. I get why we're doing this, but having to update this with each release seems like it will be a future headache. |
||
| or run to pull from docker-compose.yml: `docker compose pull` | ||
| 5. Run all collectors: `docker compose up -d` | ||
| or run a single one: `docker compose up -d scheduler-jamf` | ||
|
|
||
| Full configuration reference: https://bloodhound.specterops.io/openhound/enterprise | ||
|
|
||
| ## WARNING: | ||
| All config and secret files referenced below MUST exist before running | ||
| `docker compose up`. If they are missing, Docker will create them as directories, | ||
| which will cause the collector to fail. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I talked with @d3vzer0 and he said we should be able to keep this in the config.toml if it's the same for all collectors. I think that will be the default for most customers, so let's move it there to avoid having to set it in multiple places.