How to Export Logs¶
This guide covers extracting Silo audit logs using the SDK and the scripts/export_logs.py standalone script.
The Silo Logs Reference Guide on the Authentic8 support portal describes each log type and its fields.
Two approaches
- SDK directly — Best for programmatic integration and custom processing
- export_logs.py script — Best for one-time exports, scheduled exports, and resume support
Using the SDK Directly¶
1. Check what's available¶
from silo_sdk import LogExtractionAPI, load_config
config = load_config()
logs = LogExtractionAPI(config)
seq_info = logs.get_log_sequence_info("my_org")
print(f"Start from seq: {seq_info['min_seq']}")
print(f"More logs available: {seq_info['is_more']}")
# Note: max_seq is always None — the API does not expose total log count
print(f"Valid types: {LogExtractionAPI.VALID_LOG_TYPES}")
VALID_LOG_TYPES is compiled into the SDK, so it reflects the platform as of the SDK
release. To ask the platform you are actually talking to, call get_log_types():
info = logs.get_log_types()
print(info["log_types"]) # e.g. ['A8SS', 'ADMIN_AUDIT', 'APP_LAUNCH', ...]
print(info["version"]) # date-stamped version of the list, e.g. 20260817
Names come back in SDK form, with underscores, so they can go straight into
extract_logs(). The four types that use spaces on the wire — BLOCKED URL,
CASE MANAGER, LOCATION CHANGE, POST DATA — are translated for you.
Don't use version as a cache key yet
get_log_types() accepts a version argument as a change check, but two
deployments exposing different type lists have been seen reporting the same
version. Call it without version until that is fixed, or you may miss a newly
added type.
2. Extract logs¶
# Extract AUTH and SESSION logs starting from min_seq
records = logs.extract_logs(
org="my_org",
start_seq=seq_info.get("min_seq", 0),
log_types=["AUTH", "SESSION"],
limit=500,
)
print(f"Got {len(records['logs'])} records")
if records["is_more"]:
print(f"Next page starts at seq {records['next_seq']}")
extract_logs() returns a dictionary with logs, is_more, and next_seq. Pass next_seq
as the next call's start_seq to page forward.
2a. Extract several types when one may fail¶
A combined extract_logs() request fails as a whole if any one type is unavailable — most
often ENC on an organization without encryption — even though the other types would succeed.
extract_logs_per_type() makes one request per type and merges the pages. A type that fails is
reported in type_errors instead of failing the call; the call raises only if every type fails.
page = logs.extract_logs_per_type(
org="my_org",
start_seq=seq_info.get("min_seq", 0),
log_types=["AUTH", "SESSION", "ENC"],
limit=500,
)
print(f"Got {len(page['logs'])} records")
for log_type, error in page.get("type_errors", {}).items():
print(f"{log_type} failed: {error}")
Two things to plan for when you page through the merged result:
- Deduplicate by
seq_id. Each type paginates independently, so the mergednext_seqis the lowest resume point among the types that still reportis_more. Resuming there skips no rows, but the next page can repeat rows of types that were further ahead. When no type reports more,next_seqis the lowestnext_seqany type reported, so a poller still advances. - Failed types are not covered by
next_seq. A type listed intype_errorsreturned nothing, andis_moresays nothing about it. Re-request failed types from the originalstart_seq; resuming them from the mergednext_seqwould skip their rows.
If you make the single-type requests yourself (for example, to add your own retry), pass the
per-type results to merge_per_type_extracts() from silo_sdk.logging to get the same merged
page.
3. Export to file¶
CSV output uses per-type field schemas from log_schemas.py to produce consistent column ordering.
4. Extract all logs (with automatic pagination)¶
from datetime import datetime
all_records = logs.extract_all_logs(
org="my_org",
start_seq=0,
log_types=["AUTH", "URL"],
start_date=datetime(2026, 1, 1),
end_date=datetime(2026, 1, 31),
)
print(f"Total: {len(all_records)} records")
start_date and end_date take a datetime or an integer number of days ago; date strings
are not accepted.
Using the export_logs.py Script¶
The standalone script adds resume support, ENC log handling, and a proxy option.
Basic usage¶
source .venv/bin/activate
export A8_LOG_TOKEN="your-log-token"
export A8_TOP_ORG="my_org"
# Check sequence info and valid log types
python scripts/export_logs.py --info
# Extract all logs to JSON
python scripts/export_logs.py --output logs.json --format json
# Extract specific types to CSV
python scripts/export_logs.py --log-types AUTH SESSION --output auth_session.csv --format csv
Resume support¶
The script saves the last extracted sequence to logextract_resume.json. Re-running picks up where it left off:
# First run: extracts from seq 0
python scripts/export_logs.py --output logs.json
# Second run: continues from last seq
python scripts/export_logs.py --output logs.json
Proxy support¶
CSV enhancements¶
When exporting to CSV, you can expand JSON-encoded fields and parse hierarchy strings into separate columns:
# Expand JSON headers into individual columns (headers.User-Agent, etc.)
python scripts/export_logs.py --log-types URL --format csv --expand-json
# Flatten nested dict fields (egress_info.protocol, task_params.request_type)
python scripts/export_logs.py --log-types HARVEST --format csv --expand-nested
# Parse LAUNCHER egress_region into world/region/country/city columns
python scripts/export_logs.py --log-types LAUNCHER --format csv --parse-egress
# Combine all enhancements
python scripts/export_logs.py --log-types ALL --format csv \
--expand-json --expand-nested --parse-egress
Encrypted logs¶
The SDK handles ENC log extraction and decryption natively — no external package required:
python scripts/export_logs.py --encrypted --decrypt --pvtkey pvtkey.txt
python scripts/export_logs.py --encrypted --show-enc # include raw ENC blocks
For a standalone example of the full extract-decrypt workflow, see examples/decrypt_logs.py:
# Extract and decrypt ENC logs, print per-type breakdown
python examples/decrypt_logs.py --org my_org --pvtkey pvtkey.txt --limit 100
# Write decrypted logs to JSON
python examples/decrypt_logs.py --org my_org --pvtkey pvtkey.txt --output decrypted.json
Copy examples/pvtkey.txt.example to pvtkey.txt and fill in your private key values. See the file for format details (Legacy passphrase and Standard EC PEM formats).
Schema Validation¶
Validate the SDK's log_schemas.py field definitions against real log data from a live org:
# Validate all 26 log types against ExampleOrg
python scripts/dev/validate_log_schemas.py --org ExampleOrg
# Validate specific types only
python scripts/dev/validate_log_schemas.py --org ExampleOrg --types AUTH,SESSION,URL
# Write report to file
python scripts/dev/validate_log_schemas.py --org ExampleOrg --output validation_report.txt
The script compares LOG_TYPE_FIELDS schemas against real extracted log fields and reports any extra fields in the real data that should be added to the schema. Run this after Authentic8 ships a new logtype_headers.ini to catch schema drift.
Log type coverage
Log availability depends on org activity; not all types may have data in every organization.
Multi-Type Extraction¶
For extracting multiple log types with automatic lock-contention retry, use scripts/extract_all_logs.py:
# Extract all log types for the last 7 days
python scripts/extract_all_logs.py --org my_org --types ALL --days 7
# Extract specific types with custom limits
python scripts/extract_all_logs.py --org my_org --types AUTH,SESSION,URL --days 30 --limit 500
# Custom retry and pause settings
python scripts/extract_all_logs.py --org my_org --types ALL \
--pause 15 --retry-delay 90 --max-retries 5
The script extracts one type at a time with configurable pauses between types, and automatically retries on backend lock contention errors.
Log Analytics¶
After extracting logs, use scripts/log_stats.py to generate analytics:
# Domain frequency analysis
python scripts/log_stats.py --input logs/ --analysis domains --top 20
# Authentication pattern analysis
python scripts/log_stats.py --input logs/ --analysis auth
# Run all analyses
python scripts/log_stats.py --input logs/ --analysis all
# Output as JSON or CSV
python scripts/log_stats.py --input logs/ --analysis domains --format json
python scripts/log_stats.py --input logs/ --analysis egress --format csv
Available analyses: domains, auth, users, egress, transfers, harvest, apps, nexus, sessions, bypasses, user_agents.
Log Parsing Utilities¶
The silo_sdk.logging.log_utils module provides helpers for working with extracted log data:
from silo_sdk.logging import (
parse_json_field,
parse_all_json_fields,
extract_user_agents,
extract_content_types,
parse_egress_hierarchy,
normalize_log_types,
expand_json_fields,
flatten_nested_dicts,
)
# Parse JSON-encoded header fields
entry = parse_all_json_fields(url_log_entry)
print(entry["headers"]["User-Agent"])
# Get User-Agent frequency table
ua_counts = extract_user_agents(url_logs)
print(ua_counts.most_common(5))
# Parse LAUNCHER egress hierarchy
location = parse_egress_hierarchy(
"World / / North America / / United States / / New York, NY"
)
# {'world': 'World', 'region': 'North America',
# 'country': 'United States', 'city': 'New York, NY'}
Wire Format Note¶
Space-delimited log types
Four log types use spaces on the wire: BLOCKED URL, CASE MANAGER,
LOCATION CHANGE, POST DATA. The SDK accepts both forms — pass
BLOCKED_URL (with underscores) and the SDK converts to spaces
automatically when sending to the API.
Log Types Reference¶
The SDK ships with 26 log types; a newer deployment may support more, which is what
get_log_types() is for. Common ones:
| Type | Description |
|---|---|
AUTH |
Login/logout events |
SESSION |
Silo session lifecycle |
URL |
Web requests from Silo sessions |
EXPLOIT |
Security exploit detections |
CLIPBOARD |
Clipboard copy/paste events |
HARVEST |
Web harvesting task events |
APP_LAUNCH |
Application launch events |
EVENT |
General platform events |
ENC |
Encrypted log entries (requires decryption) |
NEXUS |
Nexus AI conversation events |
For the full list, ask the platform with LogExtractionAPI.get_log_types(), or read the
built-in LogExtractionAPI.VALID_LOG_TYPES if you need an answer offline.