How to Export Logs¶

This guide covers extracting Silo audit logs using the SDK and the scripts/export_logs.py standalone script.

Two approaches

SDK directly — Best for programmatic integration and custom processing
export_logs.py script — Best for one-time exports, scheduled exports, and resume support

Using the SDK Directly¶

1. Check what's available¶

from silo_sdk import LogExtractionAPI, load_config

config = load_config("config/default.json")
logs = LogExtractionAPI(config)

seq_info = logs.get_log_sequence_info("my_org")
print(f"Start from seq: {seq_info['min_seq']}")
print(f"More logs available: {seq_info['is_more']}")
# Note: max_seq is always None — the API does not expose total log count
print(f"Valid types: {LogExtractionAPI.VALID_LOG_TYPES}")

2. Extract logs¶

# Extract AUTH and SESSION logs starting from min_seq
records = logs.extract_logs(
    org="my_org",
    start_seq=seq_info.get("min_seq", 0),
    log_types=["AUTH", "SESSION"],
    limit=500,
)
print(f"Got {len(records)} records")

3. Export to file¶

JSONCSV

logs.export_logs_to_file(
    org="my_org",
    output_path="logs.json",
    log_types=["AUTH", "SESSION", "URL"],
    output_format="json",
)

logs.export_logs_to_file(
    org="my_org",
    output_path="logs.csv",
    log_types=["AUTH"],
    output_format="csv",
)

CSV output uses per-type field schemas from log_schemas.py to produce consistent column ordering.

4. Extract all logs (with automatic pagination)¶

all_records = logs.extract_all_logs(
    org="my_org",
    log_types=["AUTH", "URL"],
    start_date="2026-01-01",
    end_date="2026-01-31",
)
print(f"Total: {len(all_records)} records")

Using the export_logs.py Script¶

The standalone script adds resume support, ENC log handling, and a proxy option.

Basic usage¶

source .venv/bin/activate
export A8_LOG_TOKEN="your-log-token"
export A8_TOP_ORG="my_org"

# Check sequence info and valid log types
python scripts/export_logs.py --info

# Extract all logs to JSON
python scripts/export_logs.py --output logs.json --format json

# Extract specific types to CSV
python scripts/export_logs.py --log-types AUTH SESSION --output auth_session.csv --format csv

Resume support¶

The script saves the last extracted sequence to logextract_resume.json. Re-running picks up where it left off:

# First run: extracts from seq 0
python scripts/export_logs.py --output logs.json

# Second run: continues from last seq
python scripts/export_logs.py --output logs.json

Proxy support¶

python scripts/export_logs.py --proxy https://proxy.company.com:8080

CSV enhancements¶

When exporting to CSV, you can expand JSON-encoded fields and parse hierarchy strings into separate columns:

# Expand JSON headers into individual columns (headers.User-Agent, etc.)
python scripts/export_logs.py --log-types URL --format csv --expand-json

# Flatten nested dict fields (egress_info.protocol, task_params.request_type)
python scripts/export_logs.py --log-types HARVEST --format csv --expand-nested

# Parse LAUNCHER egress_region into world/region/country/city columns
python scripts/export_logs.py --log-types LAUNCHER --format csv --parse-egress

# Combine all enhancements
python scripts/export_logs.py --log-types ALL --format csv \
    --expand-json --expand-nested --parse-egress

Encrypted logs¶

The SDK handles ENC log extraction and decryption natively — no external package required:

python scripts/export_logs.py --encrypted --decrypt --pvtkey pvtkey.txt
python scripts/export_logs.py --encrypted --show-enc  # include raw ENC blocks

For a standalone example of the full extract-decrypt workflow, see examples/decrypt_logs.py:

# Extract and decrypt ENC logs, print per-type breakdown
python examples/decrypt_logs.py --org my_org --pvtkey pvtkey.txt --limit 100

# Write decrypted logs to JSON
python examples/decrypt_logs.py --org my_org --pvtkey pvtkey.txt --output decrypted.json

Copy examples/pvtkey.txt.example to pvtkey.txt and fill in your private key values. See the file for format details (Legacy passphrase and Standard EC PEM formats).

Schema Validation¶

Validate the SDK's log_schemas.py field definitions against real log data from a live org:

# Validate all 26 log types against ExampleOrg
python scripts/validate_log_schemas.py --org ExampleOrg

# Validate specific types only
python scripts/validate_log_schemas.py --org ExampleOrg --types AUTH,SESSION,URL

# Write report to file
python scripts/validate_log_schemas.py --org ExampleOrg --output validation_report.txt

The script compares LOG_TYPE_FIELDS schemas against real extracted log fields and reports any extra fields in the real data that should be added to the schema. Run this after Authentic8 ships a new logtype_headers.ini to catch schema drift.

Log type coverage

Log availability depends on org activity; not all types may have data in every organization.

Multi-Type Extraction¶

For extracting multiple log types with automatic lock-contention retry, use scripts/extract_all_logs.py:

# Extract all log types for the last 7 days
python scripts/extract_all_logs.py --org my_org --types ALL --days 7

# Extract specific types with custom limits
python scripts/extract_all_logs.py --org my_org --types AUTH,SESSION,URL --days 30 --limit 500

# Custom retry and pause settings
python scripts/extract_all_logs.py --org my_org --types ALL \
    --pause 15 --retry-delay 90 --max-retries 5

The script extracts one type at a time with configurable pauses between types, and automatically retries on backend lock contention errors.

Log Analytics¶

After extracting logs, use scripts/log_stats.py to generate analytics:

# Domain frequency analysis
python scripts/log_stats.py --input logs/ --analysis domains --top 20

# Authentication pattern analysis
python scripts/log_stats.py --input logs/ --analysis auth

# Run all analyses
python scripts/log_stats.py --input logs/ --analysis all

# Output as JSON or CSV
python scripts/log_stats.py --input logs/ --analysis domains --format json
python scripts/log_stats.py --input logs/ --analysis egress --format csv

Available analyses: domains, auth, users, egress, transfers, harvest, apps, nexus, sessions, bypasses, user_agents.

Log Parsing Utilities¶

The silo_sdk.logging.log_utils module provides helpers for working with extracted log data:

from silo_sdk.logging import (
    parse_json_field,
    parse_all_json_fields,
    extract_user_agents,
    extract_content_types,
    parse_egress_hierarchy,
    normalize_log_types,
    expand_json_fields,
    flatten_nested_dicts,
)

# Parse JSON-encoded header fields
entry = parse_all_json_fields(url_log_entry)
print(entry["headers"]["User-Agent"])

# Get User-Agent frequency table
ua_counts = extract_user_agents(url_logs)
print(ua_counts.most_common(5))

# Parse LAUNCHER egress hierarchy
location = parse_egress_hierarchy(
    "World /  / North America /  / United States /  / New York, NY"
)
# {'world': 'World', 'region': 'North America',
#  'country': 'United States', 'city': 'New York, NY'}

Wire Format Note¶

Space-delimited log types

Four log types use spaces on the wire: BLOCKED URL, CASE MANAGER, LOCATION CHANGE, POST DATA. The SDK accepts both forms — pass BLOCKED_URL (with underscores) and the SDK converts to spaces automatically when sending to the API.

Log Types Reference¶

The SDK supports 26 log types. Common ones:

Type	Description
`AUTH`	Login/logout events
`SESSION`	Silo session lifecycle
`URL`	Web requests from Silo sessions
`EXPLOIT`	Security exploit detections
`CLIPBOARD`	Clipboard copy/paste events
`HARVEST`	Web harvesting task events
`APP_LAUNCH`	Application launch events
`EVENT`	General platform events
`ENC`	Encrypted log entries (requires decryption)
`NEXUS`	Nexus AI conversation events

For the full list: LogExtractionAPI.VALID_LOG_TYPES