How to Bulk Harvest¶
Use HarvesterAPI to create web harvest tasks (screenshots, asset downloads, video capture) and retrieve the results.
Use async for bulk operations
For harvesting multiple URLs, use the async methods shown below to run tasks concurrently. This is significantly faster than processing URLs one at a time.
Task Types¶
task_type |
Description | Params key |
|---|---|---|
"visual" |
Full-page screenshot, PDF, or MHTML capture | vis_params |
"asset" |
Recursive file/asset download via wget | wget_params |
"video" |
Video download and transcription via yt-dlp | vid_params |
"single" |
Single-URL non-recursive fetch; same wget_params as asset; available from all egress locations |
wget_params |
"wget" |
Accepted as its own type and run as an asset task, so it takes the same parameters and egress limits as asset |
wget_params |
See the Web Harvesting API reference for all supported task_params keys per type.
Single Task (Synchronous)¶
from silo_sdk import HarvesterAPI, load_config
config = load_config()
harvester = HarvesterAPI(config)
# Visual capture — full-page image
task_id = harvester.create_harvest_task(
task_type="visual",
urls=["https://example.com"],
egress_info={"name": "New York, NY"},
dest_path="/screenshots/",
task_params={"output_image": True},
)
print(f"Task created: {task_id}")
# Poll until done
result = harvester.wait_for_completion(task_id, max_wait_time=300, check_interval=5)
if result and result.get("status") == "done":
print("Capture complete")
elif result and result.get("status") == "error":
print(f"Failed: {result.get('last_error')}")
Single Task (Async)¶
import asyncio
from silo_sdk import HarvesterAPI, load_config
config = load_config()
harvester = HarvesterAPI(config)
async def harvest_one(url: str) -> dict:
task_id = harvester.create_harvest_task(
task_type="visual",
urls=[url],
egress_info={"name": "New York, NY"},
dest_path="/screenshots/",
task_params={"output_pdf": True, "output_image": True},
)
return await harvester.wait_for_completion_async(
task_id, check_interval=5, max_wait_time=300
)
result = asyncio.run(harvest_one("https://example.com"))
print(result.get("status"))
Bulk Tasks (Async Concurrent)¶
Use async for bulk harvesting — tasks run concurrently:
import asyncio
from silo_sdk import HarvesterAPI, load_config
config = load_config()
harvester = HarvesterAPI(config)
urls = [
"https://site1.example.com",
"https://site2.example.com",
"https://site3.example.com",
]
async def harvest_all(urls: list[str]) -> None:
# Create all tasks first
task_ids = []
for url in urls:
task_id = harvester.create_harvest_task(
task_type="visual",
urls=[url],
egress_info={"name": "New York, NY"},
dest_path="/screenshots/",
task_params={"output_image": True},
)
task_ids.append(task_id)
print(f"Created: {task_id} → {url}")
# Poll all tasks concurrently
results = await asyncio.gather(*[
harvester.wait_for_completion_async(task_id, check_interval=5, max_wait_time=300)
for task_id in task_ids
])
for task_id, result in zip(task_ids, results):
status = result.get("status") if result else "timeout"
print(f"Task {task_id}: {status}")
asyncio.run(harvest_all(urls))
Scheduled Daily Captures (run_after)¶
Use run_after to queue tasks at a future time. Accepts a Unix epoch integer or ISO 8601 UTC string:
import time
task_id = harvester.create_harvest_task(
task_type="visual",
urls=["https://example.com"],
egress_info={"name": "London"},
dest_path="/daily/",
task_params={"output_pdf": True},
run_after=int(time.time()) + 3600, # 1 hour from now
)
For automated daily scheduling with jitter, use the scheduled_visual_harvest.py script:
# 7 days starting now, London egress, screenshot + image
python scripts/scheduled_visual_harvest.py https://example.com \
--egress London --output-image
# PDF capture, A4 landscape, specific start time
python scripts/scheduled_visual_harvest.py https://target.com \
--egress "New York, NY" --start 2026-03-01T09:00:00Z \
--days 7 --dest-path /daily-captures/ \
--output-pdf --paper A4 --landscape
# Preview the schedule without submitting
python scripts/scheduled_visual_harvest.py https://example.com \
--egress London --output-image --dry-run
# Any egress location, chosen by the platform
python scripts/scheduled_visual_harvest.py https://example.com \
--egress World --output-image
--egress accepts any egress name: World (any location), a region (for example
Europe), a country (Germany) or a city (London). Run the script with
--list-egress to print them grouped under those headings.
Tasks enter wait status until their scheduled time, then activate automatically.
Task Status Values¶
| Status | Meaning |
|---|---|
wait |
Queued but not yet started (includes run_after deferred tasks) |
claimed |
Assigned to a Silo node, and executing |
done |
Completed successfully |
error |
Failed — check last_error field |
These four are the complete set. There is no separate running status — a
task that is actively executing reads claimed, so code branching on
"running" never matches. There is also no progress percentage; status
and finished are the only signals the platform reports.
Asset Download Example¶
task_id = harvester.create_harvest_task(
task_type="asset",
urls=["https://example.com"],
egress_info={"name": "New York, NY", "categories": {"connectivity": "datacenter"}},
dest_path="/assets/",
task_params={
"recursive": "true",
"level": "2",
"page-requisites": "true",
"quota": "100m",
},
)
Egress Locations¶
egress_info={"name": "New York, NY"}
egress_info={"name": "London"}
egress_info={"name": "Dallas, TX"}
# Asset and wget tasks: one of the six datacenter locations
egress_info={
"name": "New York, NY",
"categories": {"connectivity": "datacenter", "availability": "public"},
}
Use BrowsingAPI.get_egress_locations() to list available location names.
For enriched metadata including connectivity type, availability, and supported protocols per city,
call BrowsingAPI.get_egress_locations(include_details=True):
from silo_sdk.browsing import BrowsingAPI
locations = BrowsingAPI.get_egress_locations(include_details=True)
# Returns: {"North America": [{"name": "New York, NY", "connectivity": ["datacenter", "isp", "wireless-carrier"],
# "availability": ["public"], "protocol": ["direct"]}, ...], ...}
The lists in that reply are the set of values a city offers — pick one of each when you
build categories, which takes scalars.
get_egress_locations() is a static table that ships with the SDK, so it cannot know what
your organization is actually entitled to. To ask the platform instead, use
list_egresses():
from silo_sdk import BrowsingAPI
browsing = BrowsingAPI(config)
available = browsing.list_egresses(username="analyst@example.com")
# [{"name": "New York, NY", "path": ["North America", "United States"],
# "categories": [{"connectivity": "datacenter", "availability": "public",
# "protocol": "direct"}, ...]}, ...]
This returns only the locations that are up and permitted by your organization's policy.
An empty list is a real answer: it means no egress can be pinned, usually because the
organization uses Traffic Manager rules that choose the egress themselves. It needs an
ADMIN_TOKEN.
To check egress before a large run, call harvester.list_harvest_egresses(task_type) once per task type, or set LIVE_EGRESS_VALIDATION=true; the lookup is reused for a minute per pool, so a bulk run asks once. A failed lookup is reused the same way, so it is logged once rather than per task.
TOR Routing¶
TOR egress is available for visual, video, and single task types only (not asset or wget).
Two cities support TOR: Sydney (direct + TOR) and Moncks Corner, SC (TOR only).
# TOR routing — random exit node (recommended)
egress_info={"name": "World", "categories": {"protocol": "tor"}}
# TOR routing — constrained to a specific TOR-capable city
egress_info={"name": "Sydney", "categories": {"protocol": "tor"}}
egress_info={"name": "Moncks Corner, SC", "categories": {"protocol": "tor"}}
# Geographic hierarchy also works (resolves to nearest TOR-capable city)
egress_info={"name": "Australia", "categories": {"protocol": "tor"}} # → Sydney
egress_info={"name": "United States", "categories": {"protocol": "tor"}} # → Moncks Corner
Sovereign Routing¶
"sovereign" routes through an in-country egress, for work with data-residency
requirements. It is provisioned per organization, and only for toolbox-standalone
(Silo for Research) sessions. World with protocol: "sovereign" lets the platform
choose the location:
Because provisioning is per organization, the SDK cannot cross-check the location for
you the way it does for tor — it accepts the value and lets the platform decide. To
find your organization's sovereign locations, call
list_egresses(..., session_type="toolbox-standalone") and look for
"protocol": "sovereign" in a location's categories. Expect an error from the
platform if the combination is not provisioned.
Download Results¶
Harvest results are stored in the bucket specified by BUCKET_ID in your config (or override per-task with dest_bucket_id). Download with FileAPI: