Skip to content

Web Harvesting API

Create and manage web harvest tasks (screenshots, scraping). Supports async polling and bulk operations. Requires SCRAPE_TOKEN.

Bases: BaseAPIClient

API client for Silo web harvesting operations.

This class provides methods for creating and managing automated web content harvesting tasks using the Authentic8 Silo platform. The harvester can collect various types of web content including assets, visual captures, and videos.

The harvesting API allows you to: - Create asset collection tasks (files, documents, media) - Generate visual captures (screenshots, PDFs, MHTML) - Record video captures of web interactions - Monitor task progress and status - Download completed harvest results - Manage task lifecycle and cleanup

Note

The ext API has no command to list or enumerate harvest tasks. To retrieve task IDs after the fact, extract logs of type HARVEST using LogExtractionAPI and parse the entries for task IDs, status, and output file references. The recommended pattern is to store task IDs locally at creation time via create_harvest_task().

Supported harvest types: - Asset: Download files/pages via wget. Limited to 6 egress locations (Singapore, Dubai, Frankfurt, Sao Paulo, Johannesburg, New York, NY). - Wget: Resolved as Asset by the harvest worker; same parameters and egress limits. - Single: Single-URL fetch available from all egress locations; uses wget_params. - Visual: Capture screenshots, generate PDFs, save MHTML archives (all egress). - Video: Record video of web page interactions and navigation (all egress).

Attributes:

Name Type Description
VALID_TASK_TYPES frozenset[str]

Valid task type strings. Canonical definition is silo_sdk.utils.harvester_utils.VALID_TASK_TYPES.

VALID_VISUAL_PARAMS set[str]

Set of valid visual task parameter keys including:

  • output_pdf: Generate PDF output (default: screenshot PNG)
  • output_native_pdf: Generate a native (non-screenshot) PDF
  • conform_page_to_paper: Fit page content to paper size
  • output_mhtml: Generate MHTML archive output
  • output_image: Generate screenshot image (default format)
  • scale: Rendering scale factor (0.1-2.0)
  • landscape: Landscape orientation for PDF (boolean)
  • paper: Paper size for PDF (e.g., "A4", "Letter")
  • max_scroll: Maximum scroll depth for full-page capture
  • page_strategy: Page load strategy ("eager", "complete")
  • wait_nav: Navigation wait condition (takes a value, e.g. "load"; passing True makes the worker error)
  • chrome_stealth: Enable stealth mode to reduce bot detection
  • xff: X-Forwarded-For value to send (takes a value; passing True makes the worker error)
  • translation_only: Only apply translation, skip other processing
  • cookies_out: Export session cookies after capture
VALID_ASSET_PARAMS set[str]

Set of valid asset task parameter keys (wget-style options). Sent as wget_params in the wire format. Note: cookies and return-cookies are top-level task_params fields, not wget flags, so they are not in this set.

VALID_VIDEO_PARAMS set[str]

Set of valid video task parameter keys (yt-dlp-style options). Sent as vid_params in the wire format. Key corrections: use referer (not referrer) and merge-output-format (not merge-outputformat).

Examples:

>>> from silo_sdk import HarvesterAPI, load_config
>>> config = load_config()
>>> harvester = HarvesterAPI(config)
>>>
>>> # Create an asset harvesting task
>>> task_id = harvester.create_harvest_task(
...     task_type="asset",
...     urls=["https://example.com/document.pdf"],
...     egress_info={"name": "New York, NY"},
...     dest_path="/downloads/",
...     dest_name="document.pdf",
...     task_params={"recursive": True, "level": 2}
... )
>>>
>>> # Create task with specific egress categories
>>> task_id = harvester.create_harvest_task(
...     task_type="visual",
...     urls=["https://example.com/page"],
...     egress_info={
...         "name": "Singapore",
...         "categories": {
...             "connectivity": "datacenter",
...             "availability": None,
...             "protocol": "tor"
...         }
...     },
...     task_params={"output_pdf": True}
... )
>>>
>>> # Monitor task until completion
>>> result = harvester.wait_for_completion(task_id)
>>>
>>> # Download the result
>>> harvester.download_task_result(task_id, "/local/path/document.pdf")
Source code in silo_sdk/harvesting/harvester_api.py
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
 299
 300
 301
 302
 303
 304
 305
 306
 307
 308
 309
 310
 311
 312
 313
 314
 315
 316
 317
 318
 319
 320
 321
 322
 323
 324
 325
 326
 327
 328
 329
 330
 331
 332
 333
 334
 335
 336
 337
 338
 339
 340
 341
 342
 343
 344
 345
 346
 347
 348
 349
 350
 351
 352
 353
 354
 355
 356
 357
 358
 359
 360
 361
 362
 363
 364
 365
 366
 367
 368
 369
 370
 371
 372
 373
 374
 375
 376
 377
 378
 379
 380
 381
 382
 383
 384
 385
 386
 387
 388
 389
 390
 391
 392
 393
 394
 395
 396
 397
 398
 399
 400
 401
 402
 403
 404
 405
 406
 407
 408
 409
 410
 411
 412
 413
 414
 415
 416
 417
 418
 419
 420
 421
 422
 423
 424
 425
 426
 427
 428
 429
 430
 431
 432
 433
 434
 435
 436
 437
 438
 439
 440
 441
 442
 443
 444
 445
 446
 447
 448
 449
 450
 451
 452
 453
 454
 455
 456
 457
 458
 459
 460
 461
 462
 463
 464
 465
 466
 467
 468
 469
 470
 471
 472
 473
 474
 475
 476
 477
 478
 479
 480
 481
 482
 483
 484
 485
 486
 487
 488
 489
 490
 491
 492
 493
 494
 495
 496
 497
 498
 499
 500
 501
 502
 503
 504
 505
 506
 507
 508
 509
 510
 511
 512
 513
 514
 515
 516
 517
 518
 519
 520
 521
 522
 523
 524
 525
 526
 527
 528
 529
 530
 531
 532
 533
 534
 535
 536
 537
 538
 539
 540
 541
 542
 543
 544
 545
 546
 547
 548
 549
 550
 551
 552
 553
 554
 555
 556
 557
 558
 559
 560
 561
 562
 563
 564
 565
 566
 567
 568
 569
 570
 571
 572
 573
 574
 575
 576
 577
 578
 579
 580
 581
 582
 583
 584
 585
 586
 587
 588
 589
 590
 591
 592
 593
 594
 595
 596
 597
 598
 599
 600
 601
 602
 603
 604
 605
 606
 607
 608
 609
 610
 611
 612
 613
 614
 615
 616
 617
 618
 619
 620
 621
 622
 623
 624
 625
 626
 627
 628
 629
 630
 631
 632
 633
 634
 635
 636
 637
 638
 639
 640
 641
 642
 643
 644
 645
 646
 647
 648
 649
 650
 651
 652
 653
 654
 655
 656
 657
 658
 659
 660
 661
 662
 663
 664
 665
 666
 667
 668
 669
 670
 671
 672
 673
 674
 675
 676
 677
 678
 679
 680
 681
 682
 683
 684
 685
 686
 687
 688
 689
 690
 691
 692
 693
 694
 695
 696
 697
 698
 699
 700
 701
 702
 703
 704
 705
 706
 707
 708
 709
 710
 711
 712
 713
 714
 715
 716
 717
 718
 719
 720
 721
 722
 723
 724
 725
 726
 727
 728
 729
 730
 731
 732
 733
 734
 735
 736
 737
 738
 739
 740
 741
 742
 743
 744
 745
 746
 747
 748
 749
 750
 751
 752
 753
 754
 755
 756
 757
 758
 759
 760
 761
 762
 763
 764
 765
 766
 767
 768
 769
 770
 771
 772
 773
 774
 775
 776
 777
 778
 779
 780
 781
 782
 783
 784
 785
 786
 787
 788
 789
 790
 791
 792
 793
 794
 795
 796
 797
 798
 799
 800
 801
 802
 803
 804
 805
 806
 807
 808
 809
 810
 811
 812
 813
 814
 815
 816
 817
 818
 819
 820
 821
 822
 823
 824
 825
 826
 827
 828
 829
 830
 831
 832
 833
 834
 835
 836
 837
 838
 839
 840
 841
 842
 843
 844
 845
 846
 847
 848
 849
 850
 851
 852
 853
 854
 855
 856
 857
 858
 859
 860
 861
 862
 863
 864
 865
 866
 867
 868
 869
 870
 871
 872
 873
 874
 875
 876
 877
 878
 879
 880
 881
 882
 883
 884
 885
 886
 887
 888
 889
 890
 891
 892
 893
 894
 895
 896
 897
 898
 899
 900
 901
 902
 903
 904
 905
 906
 907
 908
 909
 910
 911
 912
 913
 914
 915
 916
 917
 918
 919
 920
 921
 922
 923
 924
 925
 926
 927
 928
 929
 930
 931
 932
 933
 934
 935
 936
 937
 938
 939
 940
 941
 942
 943
 944
 945
 946
 947
 948
 949
 950
 951
 952
 953
 954
 955
 956
 957
 958
 959
 960
 961
 962
 963
 964
 965
 966
 967
 968
 969
 970
 971
 972
 973
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278
2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
2317
2318
2319
2320
2321
2322
2323
2324
2325
2326
2327
2328
2329
2330
2331
2332
2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
2347
2348
2349
2350
2351
2352
2353
2354
2355
2356
2357
2358
2359
2360
2361
2362
2363
2364
2365
2366
2367
2368
2369
2370
2371
2372
2373
2374
2375
2376
2377
2378
2379
2380
2381
2382
2383
2384
2385
2386
2387
2388
2389
2390
2391
2392
2393
2394
2395
2396
2397
2398
2399
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
2421
2422
2423
2424
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
2440
2441
2442
2443
2444
2445
2446
2447
2448
2449
2450
2451
2452
2453
2454
2455
2456
2457
2458
2459
2460
2461
2462
2463
2464
2465
2466
2467
2468
2469
2470
2471
2472
2473
2474
2475
2476
2477
2478
2479
2480
2481
2482
2483
2484
2485
2486
2487
2488
2489
2490
2491
2492
2493
2494
2495
2496
2497
2498
2499
2500
2501
2502
2503
2504
2505
2506
2507
2508
2509
2510
2511
2512
2513
2514
2515
2516
2517
class HarvesterAPI(BaseAPIClient):
    """API client for Silo web harvesting operations.

    This class provides methods for creating and managing automated web content
    harvesting tasks using the Authentic8 Silo platform. The harvester can collect
    various types of web content including assets, visual captures, and videos.

    The harvesting API allows you to:
    - Create asset collection tasks (files, documents, media)
    - Generate visual captures (screenshots, PDFs, MHTML)
    - Record video captures of web interactions
    - Monitor task progress and status
    - Download completed harvest results
    - Manage task lifecycle and cleanup

    Note:
        The ext API has no command to list or enumerate harvest tasks. To
        retrieve task IDs after the fact, extract logs of type ``HARVEST``
        using ``LogExtractionAPI``
        and parse the entries for task IDs, status, and output file
        references. The recommended pattern is to store task IDs locally
        at creation time via ``create_harvest_task()``.

    Supported harvest types:
    - Asset: Download files/pages via wget. Limited to 6 egress locations (Singapore,
      Dubai, Frankfurt, Sao Paulo, Johannesburg, New York, NY).
    - Wget: Resolved as Asset by the harvest worker; same parameters and egress limits.
    - Single: Single-URL fetch available from all egress locations; uses wget_params.
    - Visual: Capture screenshots, generate PDFs, save MHTML archives (all egress).
    - Video: Record video of web page interactions and navigation (all egress).

    Attributes:
        VALID_TASK_TYPES: Valid task type strings. Canonical definition is
            ``silo_sdk.utils.harvester_utils.VALID_TASK_TYPES``.
        VALID_VISUAL_PARAMS: Set of valid visual task parameter keys including:

            - ``output_pdf``: Generate PDF output (default: screenshot PNG)
            - ``output_native_pdf``: Generate a native (non-screenshot) PDF
            - ``conform_page_to_paper``: Fit page content to paper size
            - ``output_mhtml``: Generate MHTML archive output
            - ``output_image``: Generate screenshot image (default format)
            - ``scale``: Rendering scale factor (0.1-2.0)
            - ``landscape``: Landscape orientation for PDF (boolean)
            - ``paper``: Paper size for PDF (e.g., ``"A4"``, ``"Letter"``)
            - ``max_scroll``: Maximum scroll depth for full-page capture
            - ``page_strategy``: Page load strategy (``"eager"``, ``"complete"``)
            - ``wait_nav``: Navigation wait condition (takes a value, e.g.
              ``"load"``; passing ``True`` makes the worker error)
            - ``chrome_stealth``: Enable stealth mode to reduce bot detection
            - ``xff``: X-Forwarded-For value to send (takes a value; passing
              ``True`` makes the worker error)
            - ``translation_only``: Only apply translation, skip other processing
            - ``cookies_out``: Export session cookies after capture

        VALID_ASSET_PARAMS: Set of valid asset task parameter keys (wget-style options).
            Sent as ``wget_params`` in the wire format. Note: ``cookies`` and
            ``return-cookies`` are top-level ``task_params`` fields, not wget
            flags, so they are not in this set.
        VALID_VIDEO_PARAMS: Set of valid video task parameter keys (yt-dlp-style options).
            Sent as ``vid_params`` in the wire format. Key corrections: use ``referer``
            (not ``referrer``) and ``merge-output-format`` (not ``merge-outputformat``).

    Examples:
        >>> from silo_sdk import HarvesterAPI, load_config
        >>> config = load_config()
        >>> harvester = HarvesterAPI(config)
        >>>
        >>> # Create an asset harvesting task
        >>> task_id = harvester.create_harvest_task(
        ...     task_type="asset",
        ...     urls=["https://example.com/document.pdf"],
        ...     egress_info={"name": "New York, NY"},
        ...     dest_path="/downloads/",
        ...     dest_name="document.pdf",
        ...     task_params={"recursive": True, "level": 2}
        ... )
        >>>
        >>> # Create task with specific egress categories
        >>> task_id = harvester.create_harvest_task(
        ...     task_type="visual",
        ...     urls=["https://example.com/page"],
        ...     egress_info={
        ...         "name": "Singapore",
        ...         "categories": {
        ...             "connectivity": "datacenter",
        ...             "availability": None,
        ...             "protocol": "tor"
        ...         }
        ...     },
        ...     task_params={"output_pdf": True}
        ... )
        >>>
        >>> # Monitor task until completion
        >>> result = harvester.wait_for_completion(task_id)
        >>>
        >>> # Download the result
        >>> harvester.download_task_result(task_id, "/local/path/document.pdf")
    """

    # Valid parameters for different harvest types
    VALID_ASSET_PARAMS: set[str] = {
        "timeout",
        "limit-rate",
        "wait",
        "random-wait",
        "quota",
        "user",
        "password",
        "local-encoding",
        "remote-encoding",
        "http-user",
        "http-password",
        "no-cookies",
        "keep-session-cookies",
        "header",
        "max-redirect",
        "save-headers",
        "user-agent",
        "secure-protocol",
        "https-only",
        "no-check-certificate",
        "ftp-user",
        "ftp-password",
        "recursive",
        "level",
        "convert-links",
        "accept",
        "reject",
        "accept-regex",
        "reject-regex",
        "domains",
        "exclude-domains",
        "post-data",
        "execute",
        "span-hosts",
        "no-parent",
        "no-directories",
        "page-requisites",
        "adjust-extension",
        "notes",
    }

    VALID_VISUAL_PARAMS: set[str] = {
        # Mirrors the harvest worker's vis_flags allowlist. translation_proxy
        # was missing here while the worker accepted it, so the SDK rejected a
        # parameter the platform would have taken.
        "translation_proxy",
        "scale",
        "paper",
        "landscape",
        "cookies_out",
        "output_pdf",
        "output_mhtml",
        "output_image",
        "output_native_pdf",
        "conform_page_to_paper",
        "max_scroll",
        "page_strategy",
        "notes",
        "suppress_pdf_cover",
        "id_egress",
        "emulate",
        "is_mobile",
        "has_touch",
        "user_agent",
        "viewport_width",
        "timezone",
        "accept_language",
        "slice_avoid_top",
        "slice_avoid_bottom",
        "scroll_delay",
        "final_delay",
        "wait_nav",
        "translate_original_lang",
        "translate_target_lang",
        "translation_only",
        "ignore_cert_error",
        "chrome_stealth",
        "xff",
    }

    VALID_VIDEO_PARAMS: set[str] = {
        # Mirrors the harvest worker's vid_flags allowlist; see the note on
        # VALID_VISUAL_PARAMS for why translation_proxy was absent.
        "translation_proxy",
        "no-check-certificate",
        "write-description",
        "write-info-json",
        "write-all-thumbnails",
        "write-auto-sub",
        "no-call-home",
        "write-sub",
        "all-subs",
        "list-formats",
        "audio",
        "format",
        "recode-video",
        "remux-video",
        "user-agent",
        "username",
        "password",
        "referer",
        "add-header",
        "all-formats",
        "merge-output-format",
        "cookies",
        "filename",
        "transcribe",
        "transcribe-lang",
        "transcribe-text-only",
        "transcription-service",
        "translate-target-lang",
        "file-id",
        "full_path",
        "notes",
    }

    # Canonical set lives in harvester_utils so the builder and config
    # validator there share one definition; see the note beside it.
    VALID_TASK_TYPES: frozenset[str] = _VALID_TASK_TYPES

    TASK_PARAMS_TOP_LEVEL_KEYS: set[str] = {
        "return-cookies",
        "cookies",
        "all_files",
    }

    @cached_property
    def _valid_params_map(self) -> dict[str, set[str]]:
        """Get map of task types to valid parameters."""
        return {
            "asset": self.VALID_ASSET_PARAMS,
            "visual": self.VALID_VISUAL_PARAMS,
            "video": self.VALID_VIDEO_PARAMS,
            "single": self.VALID_ASSET_PARAMS,
            "wget": self.VALID_ASSET_PARAMS,
        }

    @classmethod
    def _top_level_keys_for_type(cls, task_type: str) -> set[str]:
        """Return the top-level task_params keys for the given task type.

        For video tasks, ``cookies`` goes in ``vid_params`` (yt-dlp list format),
        not at the top level. Visual, asset and single tasks take ``cookies`` at
        the top level (a Netscape cookie file content string); it is never
        placed in ``vis_params``.
        """
        if task_type == "video":
            return {"return-cookies", "all_files"}
        return cls.TASK_PARAMS_TOP_LEVEL_KEYS

    @staticmethod
    def _normalize_task_params(
        task_params: dict[str, Any] | None,
    ) -> dict[str, Any] | None:
        """Normalize Python-style aliases to wire-format keys."""
        if task_params and "return_cookies" in task_params:
            if "return-cookies" in task_params:
                raise ValidationError(
                    "Cannot specify both 'return_cookies' and 'return-cookies'; "
                    "use one or the other"
                )
            task_params = dict(task_params)
            task_params["return-cookies"] = task_params.pop("return_cookies")
        return task_params

    def __init__(self, config: dict[str, Any]) -> None:
        """Initialize the harvester API client.

        Args:
            config: Configuration dictionary containing API settings

        Raises:
            ConfigurationError: If required configuration is missing or invalid
        """
        super().__init__(config)

        # Comprehensive configuration validation
        validation_errors = validate_harvester_config(config)
        if validation_errors:
            raise ConfigurationError(
                f"Harvester configuration validation failed: {'; '.join(validation_errors)}"
            )

        # Basic required configuration (legacy support)
        validate_config_for_api(config, ["SCRAPE_TOKEN", "BUCKET_ID"])

        self.auth_token = config["SCRAPE_TOKEN"]
        self.dest_bucket_id = config["BUCKET_ID"]

        # Off by default; one directory lookup per pool per minute when on.
        self.live_egress_validation = _config_flag(
            config.get("LIVE_EGRESS_VALIDATION", False)
        )
        self._browsing: Any = None
        # None caches a failed lookup, so a bulk run warns once, not per task.
        self._directory_cache: dict[
            tuple[str, str], tuple[float, list[EgressDirectoryEntry] | None]
        ] = {}
        # The async methods reach check_harvest_egress via asyncio.to_thread,
        # so the cache is filled from several threads. Two levels of lock:
        # _directory_lock guards _pool_locks and the lazy BrowsingAPI and is
        # only held briefly; each (pool, org) has its own lock, held across
        # that pool's lookup, under which its cache entry is read and written.
        # Concurrent checks of one pool share a single request, and a slow
        # lookup for one pool never stalls checks of another. Lock order is
        # always pool lock, then _directory_lock (inside _browsing_api()).
        self._directory_lock = threading.RLock()
        self._pool_locks: dict[tuple[str, str], threading.Lock] = {}

        # Initialize task status cache if enabled
        self._status_cache_enabled, self._status_cache_ttl, self._status_cache = (
            init_status_cache(config)
        )

        # Initialize file API for downloading results
        self.file_api = FileAPI(config)
        self._ssl_context = self.create_ssl_context()

        self.logger.info("Initialized HarvesterAPI client with validated configuration")

    def _require_task_id(self, task_id: str) -> None:
        if not task_id or not isinstance(task_id, str):
            raise ValidationError("task_id must be a non-empty string")

    def __enter__(self) -> HarvesterAPI:
        """Enter context manager for resource management.

        Returns:
            Self for use in with statement
        """
        self.logger.debug("Entering HarvesterAPI context manager")
        return self

    def __exit__(
        self,
        exc_type: type[BaseException] | None,
        exc_val: BaseException | None,
        exc_tb: TracebackType | None,
    ) -> None:
        """Exit context manager and perform cleanup.

        Args:
            exc_type: Exception type if an exception occurred
            exc_val: Exception value if an exception occurred
            exc_tb: Exception traceback if an exception occurred
        """
        # Suppress unused parameter warnings - these are required by the context manager protocol
        _ = exc_type, exc_val, exc_tb
        try:
            self.cleanup_resources()
        except Exception as e:
            self.logger.warning("Error during context manager cleanup: %s", e)
        finally:
            self.logger.debug("Exited HarvesterAPI context manager")

    @api_tag(MethodType.UTILITY)
    def cleanup_resources(self) -> None:
        """Clean up resources and perform housekeeping tasks.

        This method should be called when done with the HarvesterAPI instance
        to properly clean up resources, clear caches, and close connections.
        """
        try:
            # Clear status cache
            if self._status_cache:
                cache_size = len(self._status_cache)
                self._status_cache.clear()
                if cache_size > 0:
                    self.logger.info("Cleared %d cached task statuses", cache_size)

            # Close every HTTP session this instance owns.
            self.close()

        except Exception as e:
            self.logger.error("Error during resource cleanup: %s", e)

    def _pool_lock(self, key: tuple[str, str]) -> threading.Lock:
        """The lock that serialises directory lookups for one (pool, org)."""
        with self._directory_lock:
            return self._pool_locks.setdefault(key, threading.Lock())

    def _close_browsing(self) -> None:
        """Close the BrowsingAPI the live egress check created, if any.

        Waits for any in-flight directory lookup first, so it never closes the
        client out from under one: it takes every pool lock (in a fixed order)
        and then ``_directory_lock`` — the same order a lookup uses, so the
        two cannot deadlock.
        """
        with self._directory_lock:
            pool_locks = [self._pool_locks[key] for key in sorted(self._pool_locks)]
        with contextlib.ExitStack() as held:
            for lock in pool_locks:
                held.enter_context(lock)
            with self._directory_lock:
                browsing = self._browsing
                if browsing is None:
                    return
                try:
                    browsing.close()
                finally:
                    self._browsing = None

    @api_tag(MethodType.UTILITY)
    def close(self) -> None:
        """Close every HTTP session this instance owns.

        That is its own session, the session of the FileAPI it builds for
        downloads, and the BrowsingAPI the live egress check creates on first
        use. Safe to call more than once. A failure closing one does not
        leave the others open: all three are attempted, and an error is
        raised afterwards (if more than one fails, the last error is raised
        with the earlier one chained as its context).
        """
        try:
            super().close()
        finally:
            try:
                file_api = getattr(self, "file_api", None)
                if file_api is not None:
                    file_api.close()
            finally:
                self._close_browsing()

    @api_tag(MethodType.API_COMMAND, api_command="create_harvest_task")
    def create_harvest_task(
        self,
        task_type: str,
        urls: list[str],
        egress_info: dict[str, Any],
        dest_path: str = "/",
        dest_name: str | None = None,
        dest_bucket_id: str | None = None,
        task_params: dict[str, Any] | None = None,
        run_after: int | float | str | None = None,
        validate_egress: bool | None = None,
    ) -> str:
        """Create a new harvest task for web content collection.

        Creates a harvesting task that will collect content from the specified URL
        using the given parameters and store the results in the designated location.

        Args:
            task_type: Type of harvest task. Must be one of ``"asset"``,
                ``"visual"``, ``"video"``, ``"single"``, or ``"wget"``.

                - ``"visual"`` — Full-page capture. Output format controlled by ``task_params``:

                  - Screenshot (default): no params, or ``{"output_image": True}``
                  - PDF: ``{"output_pdf": True}``
                  - MHTML: ``{"output_mhtml": True}``

                  **IMPORTANT:** ``"screenshot"``, ``"pdf"``, and ``"mhtml"`` are **not**
                  valid ``task_type`` values. Always use ``"visual"`` with ``task_params``.

                - ``"asset"`` — Recursive asset download (HTML, JS, CSS, images).
                  Control depth/filters via ``task_params`` (wget-style options).
                - ``"wget"`` — Accepted as its own type and run as an ``"asset"``
                  task, with the same parameters.
                - ``"video"`` — Video recording of page interaction (youtube-dl-style options).

            urls: List containing exactly one URL string to harvest.
            egress_info: Egress location dict with required ``"name"`` key
                (e.g., ``{"name": "New York, NY"}``). ``"asset"`` and ``"wget"``
                tasks draw from the six asset datacenter locations (Singapore,
                Dubai, Frankfurt, Sao Paulo, Johannesburg, ``"New York, NY"``):
                pass one of those or ``"World"``. Any other name, such as
                ``"London"``, logs a warning rather than raising, since a
                region or country works when an asset location sits beneath
                it. Optional ``"categories"``
                key accepts ``"connectivity"``, ``"availability"``, ``"protocol"``
                (``direct``, ``tor`` or ``sovereign``); omitted keys are not sent.
                A ``server_set`` key is removed before sending (the server
                rejects it for harvest tasks).
                For non-``"asset"`` task types, ``"connectivity"``/``"protocol"``
                are validated against what is actually available at ``"name"``
                (see ``silo_sdk.harvesting.egress_validation``). Does not
                apply to ``"asset"`` tasks. These checks are advisory. For
                the live answer, see :meth:`list_harvest_egresses` and
                ``validate_egress`` below: ``visual``, ``video`` and
                ``single`` tasks draw from the same pool as browsing
                contexts, ``asset`` and ``wget`` tasks from their own.
            dest_path: Destination path in storage (defaults to ``"/"``)
            dest_name: Name for the result file (auto-generated if not provided)
            dest_bucket_id: Destination bucket ID (uses default if not provided)
            task_params: Task-specific output parameters. Most keys are
                converted to a list of ``{"name": key}`` or
                ``{"name": key, "value": val}`` dicts sent under the
                appropriate wire key (``vis_params``, ``wget_params``,
                ``vid_params``). The following keys are instead injected
                directly at the top level of the wire ``task_params`` object:

                - ``return-cookies`` — send back cookies from the session
                - ``cookies`` — Netscape-format cookie file content (string).
                  Top-level for ``"asset"``, ``"single"`` and ``"visual"``
                  tasks; for ``"video"`` it is sent inside ``vid_params``.
                - ``all_files`` — include all fetched files alongside the main
                  content. Valid for ``"visual"``, ``"single"``, and ``"video"``
                  tasks. Raises ``ValidationError`` for ``"asset"`` tasks.
                  With ``all_files=True`` the task result is a directory id,
                  which ``download_task_result`` (a plain getfile) cannot
                  fetch; read it through the file API instead. ``False`` is
                  dropped, never sent.


                Key visual params: ``output_pdf``, ``output_mhtml``,
                ``output_image``, ``scale``, ``landscape``, ``paper``.
                See class ``Attributes`` for full param sets per task type.
            run_after: Optional schedule time — Unix epoch integer
                (e.g., ``1707849000``), float epoch, or timezone-aware
                ISO 8601 string (e.g., ``"2026-02-13T18:30:00Z"`` or
                ``"2026-02-13T13:30:00-05:00"``). Booleans and timezone-less
                strings are refused. Task enters ``"wait"``
                status until the specified time.
            validate_egress: Look the egress up in its pool's live directory
                (:meth:`check_harvest_egress`) and log a warning if it is not
                listed. Never blocks the task. ``None`` (default) uses the
                ``LIVE_EGRESS_VALIDATION`` config key.

        Note:
            Wire format parameter mappings:

            - ``task_type`` → ``task_params.request_type``
            - ``urls`` → ``task_params.urls``
            - ``task_params`` (Python dict) → list of ``{"name": key}`` or
              ``{"name": key, "value": val}`` dicts sent as:

              - ``task_params.vis_params`` for ``"visual"`` tasks
              - ``task_params.wget_params`` for ``"asset"`` tasks
              - ``task_params.vid_params`` for ``"video"`` tasks

            - ``egress_info``, ``dest_bucket_id``, ``dest_path``, ``dest_name``,
              ``dest_auth_token`` remain at the top-level command dict
            - ``run_after`` is injected at **top-level command dict** alongside
              ``egress_info``, **NOT** inside ``task_params``

            Wire format structure for visual task::

                {
                    "command": "create_harvest_task",
                    "egress_info": {"name": "New York, NY"},
                    "dest_bucket_id": "bucket_abc123",
                    "dest_path": "/",
                    "dest_name": "capture.pdf",
                    "dest_auth_token": "<file_token>",
                    "task_params": {
                        "request_type": "visual",
                        "urls": ["https://example.com"],
                        "vis_params": [
                            {"name": "output_pdf"},
                            {"name": "paper", "value": "A4"}
                        ]
                    },
                    "run_after": 1707849000
                }

        Returns:
            Task ID string for monitoring and managing the harvest task

        Raises:
            HarvesterAPIError: If task creation fails
            ValidationError: If parameters are invalid

        Examples:
            >>> # Create asset harvest task
            >>> task_id = api.create_harvest_task(
            ...     task_type="asset",
            ...     urls=["https://example.com/files/"],
            ...     egress_info={
            ...         "name": "New York, NY",
            ...         "categories": {
            ...             "connectivity": "datacenter",
            ...             "availability": "private",
            ...             "protocol": "direct"
            ...         }
            ...     },
            ...     dest_path="/downloads/",
            ...     dest_name="harvested_files.zip",
            ...     task_params={
            ...         "recursive": True,
            ...         "level": 2,
            ...         "accept": "*.pdf,*.doc,*.docx"
            ...     }
            ... )
            >>>
            >>> # Create visual PDF capture
            >>> task_id = api.create_harvest_task(
            ...     task_type="visual",  # ← NOT "pdf"
            ...     urls=["https://example.com/report"],
            ...     egress_info={"name": "Singapore"},
            ...     dest_name="report_capture.pdf",
            ...     task_params={
            ...         "output_pdf": True,  # ← controls output format
            ...         "paper": "A4",
            ...         "landscape": False
            ...     }
            ... )
            >>>
            >>> # Create visual MHTML capture
            >>> task_id = api.create_harvest_task(
            ...     task_type="visual",  # ← NOT "mhtml"
            ...     urls=["https://example.com/page"],
            ...     egress_info={"name": "New York, NY"},
            ...     task_params={"output_mhtml": True}  # ← MHTML format
            ... )
        """
        task_params = self._normalize_task_params(task_params)
        top_level_keys = self._top_level_keys_for_type(task_type)

        # Validate all input parameters
        validate_task_creation_params(
            task_type,
            urls,
            egress_info,
            dest_path,
            task_params,
            self.VALID_TASK_TYPES,
            self._valid_params_map,
            top_level_keys=top_level_keys,
        )
        if run_after is not None:
            validate_run_after(run_after)
        if self.live_egress_validation if validate_egress is None else validate_egress:
            self.check_harvest_egress(task_type, egress_info)

        # Prepare normalized data for API
        normalized_egress_info = normalize_egress_info(egress_info)
        bucket_id = dest_bucket_id or self.dest_bucket_id
        final_dest_name = generate_dest_name_if_needed(dest_name, task_type)

        # Build and execute the task creation request
        payload = build_task_creation_payload(
            task_type,
            urls,
            normalized_egress_info,
            bucket_id,
            dest_path,
            final_dest_name,
            task_params or {},
            self.file_api.auth_token,
            run_after=run_after,
            top_level_keys=top_level_keys,
        )

        return self._execute_task_creation(
            task_type, urls, normalized_egress_info, payload
        )

    def _execute_task_creation(
        self,
        task_type: str,
        urls: list[str],
        normalized_egress_info: dict[str, Any],
        payload: list[dict[str, Any]],
    ) -> str:
        """Execute the task creation API call.

        Args:
            task_type: Type of harvest task
            urls: List of URLs being harvested
            normalized_egress_info: Normalized egress information
            payload: API payload for task creation

        Returns:
            Task ID of the created task

        Raises:
            HarvesterAPIError: If task creation fails
        """
        try:
            self.logger.debug(
                "Creating %s harvest task for %d URLs with egress location: %s",
                task_type,
                len(urls),
                normalized_egress_info.get("name", "unknown"),
            )

            response = self._make_api_request("POST", "api/", self.auth_token, payload)
            result = self._extract_api_result(response)

            if isinstance(result, dict) and "task_id" in result:
                task_id: str = str(result["task_id"])
                self.logger.info("Created harvest task: %s", task_id)
                return task_id

            raise HarvesterAPIError(f"No task_id in response: {result}")

        except Exception as e:
            if isinstance(e, HarvesterAPIError):
                raise
            raise HarvesterAPIError(f"Failed to create harvest task: {e}") from e

    @api_tag(MethodType.API_COMMAND, api_command="find_harvest_task")
    def find_harvest_task(
        self,
        task_id: str,
        use_cache: bool = True,
        finished: bool | None = None,
    ) -> dict[str, Any]:
        """Retrieve the status and details of a specific harvest task.

        Gets comprehensive information about a harvest task including its current
        status, timestamps, results, and any error information. Supports caching
        to reduce API calls for frequently polled tasks.

        Args:
            task_id: The ID of the task to retrieve
            use_cache: Whether to use cached results (default: True)
            finished: If True, return only completed tasks. If False, return only
                in-progress tasks. If omitted, return regardless of completion state.

        Returns:
            The task record as the platform returns it. This method passes
            the record straight through — it does not rename or normalize
            anything — so the timestamps are ``create_ts``/``start_ts``/
            ``end_ts`` and the result payload is ``task_results``. The
            ``created_ts``/``started_ts``/``completed_ts``/``task_metadata``
            aliases exist only on the dictionaries returned by
            ``wait_for_completion()`` and ``wait_for_completion_async()``,
            which layer them on top of these fields.

            Identity and state:

            - task_id: Task identifier
            - status: One of ``wait``, ``claimed``, ``done`` or ``error``
            - finished: True once the task has been resolved, successfully or
                not. A finished task is never claimed or retried again
            - attempt_count: Attempts made so far, **counting the attempt
                currently in flight**. A task stays eligible to run while this
                is below the platform's attempt limit
            - last_error: Failure message, populated when ``status`` is
                ``error``. Also set when a task is failed for exhausting its
                retry limit
            - hostname: The worker that most recently claimed the task. This
                identifies the machine doing the harvesting, not the target
            - org_id: Organization that owns the task. Assigned by the
                platform at creation and not accepted from the caller

            Timing, as Unix epoch floats or null:

            - create_ts: When the task was created
            - start_ts: When a worker most recently claimed it. Null until
                the first claim, and reset on each subsequent attempt
            - end_ts: When the task was resolved. Null until then, and
                cleared again if the task is re-attempted
            - run_after: Earliest time the task may be claimed. Set from the
                ``run_after`` creation argument and pushed forward whenever a
                worker reschedules a retry

            What was requested:

            - request_type: Task type. The platform strips and lower-cases
                this, so it can come back in a different case than it was sent
            - target: Worker pool the task is routed to. Only workers polling
                for the same target can claim it
            - name: Caller-supplied label, echoed back unchanged
            - task_params: The per-type parameters the task was created with,
                including ``urls``. The platform reflects ``request_type`` into
                this dictionary and removes the internal log token from it
                before returning
            - dest_bucket_id: Storage bucket the results are written to
            - dest_path: Path within that bucket
            - dest_name: Name of the result object
            - egress_info: Egress selection in effect, as ``{"name": ...,
                "categories": {...}}``
            - egress_tag: Superseded egress selector, kept for older
                integrations. Null whenever ``egress_info`` is set, which is
                every task this SDK creates

            Results:

            - result_file_id: File ID of the result, populated once the task
                completes successfully
            - task_results: Raw result payload from the worker. Task metadata
                is nested inside it under the ``task-metadata`` key

            Note:
                **There is no progress field.** The platform reports a task's
                state through ``status`` and ``finished`` only; it does not
                report a percentage. An earlier version of this docstring
                listed ``progress`` "if available", and it is never available —
                anything reading it gets ``None`` every time.

                **``running`` is rarely, if ever, observed.** The documented
                statuses are ``wait``, ``claimed``, ``done`` and ``error``; a
                task in flight normally reads ``claimed``. The worker code
                does write ``running``, so keep code that branches on status
                tolerant of it.

        Raises:
            HarvesterAPIError: If task retrieval fails
            ValidationError: If task_id is invalid

        Examples:
            >>> task_info = api.find_harvest_task("task_123")
            >>> print(f"Status: {task_info['status']}")
            >>> if task_info['status'] == 'done':
            ...     print(f"Result file: {task_info['result_file_id']}")
            >>> elif task_info['status'] == 'error':
            ...     print(f"Error: {task_info['last_error']}")
        """
        self._require_task_id(task_id)

        # Check cache if enabled and requested
        # The cache is keyed by task id only, so a record cached without the
        # filter must not answer a filtered query.
        if (
            use_cache
            and finished is None
            and self._status_cache_enabled
            and self._status_cache is not None
        ):
            cached_result = get_cached_task_status(
                self._status_cache, task_id, self._status_cache_ttl
            )
            if cached_result is not None:
                self.logger.debug("Retrieved task status from cache: %s", task_id)
                return cached_result

        command: dict[str, Any] = {"command": "find_harvest_task", "task_id": task_id}
        if finished is not None:
            command["finished"] = finished
        payload = [command]

        try:
            self.logger.debug("Finding harvest task: %s", task_id)
            response = self._make_api_request("POST", "api/", self.auth_token, payload)

            # Debug logging for response structure (only in debug mode)
            self.logger.debug("Processing response for task: %s", task_id)

            # Handle different response formats more robustly
            try:
                result = self._extract_api_result(response)
            except SiloError as extract_error:
                # A reported API error is a real answer, not a parse failure.
                #
                # _extract_api_result raises SiloError for both shapes the
                # endpoint uses to signal failure: {"error": ...} in place of
                # result, and {"result": {"error": ...}} nested inside one,
                # and attaches the offending element as response_data. The Eng
                # capture has a recorded example of the first on this very
                # command ("Error: invalid auth token"). Falling back to
                # response[1] here handed that envelope back as if it were a
                # task: callers got {"error": ...} whose .get("status") is
                # None, it was written into the status cache, and
                # wait_for_completion reported "Unexpected task status: None"
                # instead of the server's message. Let it through.
                if extract_error.response_data is not None:
                    raise
                # No response_data means the shape was malformed rather than
                # an error report: fall back to direct response processing.
                self.logger.debug(
                    "Standard extraction failed, trying direct response processing: %s",
                    extract_error,
                )
                if isinstance(response, list) and len(response) > 1:
                    result = response[1]
                else:
                    raise HarvesterAPIError(
                        f"Unexpected response format: {response}"
                    ) from extract_error

            # Handle different result formats
            if isinstance(result, list) and len(result) > 0:
                task_details = result[0]
            elif isinstance(result, dict):
                task_details = result
            elif isinstance(result, str):
                # Handle string responses that might contain error information
                if "task produced no output" in result or "TypeError" in result:
                    # Preserve the original server error message without additional wrapping
                    raise HarvesterAPIError(
                        f"Task {task_id} failed with server error: {result}"
                    )
                else:
                    raise HarvesterAPIError(
                        f"Unexpected string response for task_id: {task_id} - {result}"
                    )
            else:
                raise HarvesterAPIError(
                    f"No task details found for task_id: {task_id} - "
                    f"got result type: {type(result)}"
                )

            # Ensure task_details is a dictionary
            if not isinstance(task_details, dict):
                # Handle string task details that might contain error information
                if isinstance(task_details, str) and (
                    "task produced no output" in task_details
                    or "TypeError" in task_details
                ):
                    raise HarvesterAPIError(
                        f"Task {task_id} failed with server error: {task_details}"
                    )
                else:
                    raise HarvesterAPIError(
                        f"Task details must be a dictionary, "
                        f"got {type(task_details)}: {task_details}"
                    )

            self.logger.debug("Retrieved task details for: %s", task_id)

            # Cache the result if caching is enabled
            if self._status_cache_enabled:
                cache_task_status(
                    self._status_cache, task_id, task_details, self._status_cache_ttl
                )

            return task_details

        except Exception as e:
            if isinstance(e, HarvesterAPIError):
                raise
            raise HarvesterAPIError(f"Failed to find harvest task: {e}") from e

    @api_tag(MethodType.API_COMMAND, api_command="delete_harvest_task")
    def delete_harvest_task(self, task_id: str) -> bool:
        """Delete a specific harvest task.

        Permanently removes a harvest task and its associated data. This action
        cannot be undone. If the task is still running, the worker aborts it at
        its next lock extension, not instantly.

        Args:
            task_id: The ID of the task to delete

        Returns:
            True if deletion was successful, False otherwise

        Raises:
            HarvesterAPIError: If task deletion fails
            ValidationError: If task_id is invalid

        Examples:
            >>> success = api.delete_harvest_task("task_123")
            >>> if success:
            ...     print("Task deleted successfully")
        """
        self._require_task_id(task_id)

        payload = [{"command": "delete_harvest_task", "task_id": task_id}]

        try:
            self.logger.debug("Deleting harvest task: %s", task_id)
            response = self._make_api_request("POST", "api/", self.auth_token, payload)

            result = self._extract_api_result(response)
            if isinstance(result, dict) and result.get("deleted"):
                self.logger.info("Successfully deleted harvest task: %s", task_id)
                return True

            self.logger.warning("Task deletion may have failed: %s", result)
            return False

        except Exception as e:
            if isinstance(e, HarvesterAPIError):
                raise
            raise HarvesterAPIError(f"Failed to delete harvest task: {e}") from e

    def _browsing_api(self) -> Any:
        """A BrowsingAPI on the same config, created on first use.

        Imported here: silo_sdk.browsing imports this package, so a module
        level import back would be circular.
        """
        with self._directory_lock:
            if self._browsing is None:
                from silo_sdk.browsing.isolation_api import BrowsingAPI

                self._browsing = BrowsingAPI(self.config)
            return self._browsing

    @api_tag(MethodType.CONVENIENCE, wraps=["list_egresses"])
    def list_harvest_egresses(
        self,
        task_type: str = "visual",
        *,
        org_id: str | int | None = None,
        org_name: str | None = None,
    ) -> list[EgressDirectoryEntry]:
        """List the egress a harvest task of this type can draw from.

        Harvest tasks resolve egress for the organization the harvest token
        administers, from a pool chosen by task type: ``asset`` and ``wget``
        use the asset pool, ``visual``, ``video`` and ``single`` the pool
        browsing contexts use. This asks the platform's directory for that
        pool and org.

        The org defaults to ``HARVEST_EGRESS_ORG``, or ``TOP_ORG`` when that
        is unset. Harvest resolves egress for the org the harvest token
        belongs to, and an org lookup sees only egress provisioned to that
        org and the orgs above it. When the harvest token belongs to a
        sub-org of ``TOP_ORG`` (common: egress provisioned to a workspace
        sub-org), set ``HARVEST_EGRESS_ORG`` to that sub-org; otherwise the
        lookup sees nothing and the live check is skipped.

        The answer can **under-report**: an org's Launcher Options hide
        regions from the directory but not from harvest, so a task may still
        succeed with an egress this list omits. An empty list does not mean
        harvest will fail: an org whose egress is provisioned to its
        sub-orgs lists nothing, so query the sub-org instead.

        Args:
            task_type: The harvest task type the egress is for.
            org_id: The org to report for. Defaults to
                ``HARVEST_EGRESS_ORG``, or ``TOP_ORG`` if that is unset.
            org_name: The org by name, as an alternative to ``org_id``.

        Returns:
            Directory entries, as :meth:`BrowsingAPI.list_egresses` returns.

        Raises:
            ValidationError: If ``task_type`` is not a harvest task type.
            ConfigurationError: Without ``ADMIN_TOKEN`` (the directory needs
                an admin token), or with no org and neither
                ``HARVEST_EGRESS_ORG`` nor ``TOP_ORG``.
            BrowsingAPIError: If the lookup fails.

        Examples:
            >>> pool = harvester.list_harvest_egresses("visual")
            >>> [e["name"] for e in pool]
            ['World', 'Europe', 'Germany', 'Frankfurt', ...]
        """
        if task_type not in self.VALID_TASK_TYPES:
            raise ValidationError(
                f"Invalid task_type: {task_type}. "
                f"Valid types are: {', '.join(sorted(self.VALID_TASK_TYPES))}"
            )
        if org_id is None and not org_name:
            org_name = self._harvest_egress_org() or None
            if not org_name:
                raise ConfigurationError(
                    "list_harvest_egresses needs org_id or org_name, or "
                    "HARVEST_EGRESS_ORG or TOP_ORG in the configuration (set "
                    "A8_HARVEST_EGRESS_ORG or A8_TOP_ORG)"
                )
        return cast(
            "list[EgressDirectoryEntry]",
            self._browsing_api().list_egresses(
                org_id=org_id,
                org_name=org_name,
                session_type=HARVEST_DIRECTORY_SESSION_TYPE,
                service_type=harvest_service_type(task_type),
            ),
        )

    def _harvest_egress_org(self) -> str:
        """Return the org harvest egress lookups use: HARVEST_EGRESS_ORG or TOP_ORG."""
        return str(
            self.config.get("HARVEST_EGRESS_ORG") or self.config.get("TOP_ORG") or ""
        )

    @api_tag(MethodType.CONVENIENCE, wraps=["list_egresses"])
    def check_harvest_egress(
        self, task_type: str, egress_info: dict[str, Any], *, warn: bool = True
    ) -> str | None:
        """Check a harvest egress against its pool's directory; never raises.

        Looks the egress up in :meth:`list_harvest_egresses` for
        ``HARVEST_EGRESS_ORG``, or ``TOP_ORG`` if that is unset, and logs a
        warning when it is not listed. A miss does not block the task,
        because the directory can under-report (see
        :meth:`list_harvest_egresses`). Lookups are reused for a minute per
        pool, so a batch asks once. A failed lookup is remembered for that
        minute too, so a batch logs it once instead of retrying per task.
        Safe to call from several threads: concurrent checks of one pool
        share a single lookup, and checks of different pools never wait on
        each other.

        Args:
            task_type: The harvest task type.
            egress_info: The task's egress selection.
            warn: Log the gap as a warning saying the task was not blocked.
                Pass ``False`` when the caller acts on the returned gap
                itself, for example by refusing the task. Other logs (a
                skipped lookup, an empty pool) are unaffected.

        Returns:
            A description of the gap, or None when the egress is listed or
            the check could not run (no ``ADMIN_TOKEN``, neither
            ``HARVEST_EGRESS_ORG`` nor ``TOP_ORG``, or a failed lookup — each
            logged).
        """
        if (
            task_type not in self.VALID_TASK_TYPES
            or not isinstance(egress_info, dict)
            or not egress_info.get("name")
        ):
            # Shape and type errors are create_harvest_task's to report.
            return None
        org_name = self._harvest_egress_org()
        key = (harvest_service_type(task_type), org_name)
        directory: list[EgressDirectoryEntry] | None
        with self._pool_lock(key):
            now = time.monotonic()
            cached = self._directory_cache.get(key)
            if cached is not None and now - cached[0] < _DIRECTORY_TTL_SECONDS:
                directory = cached[1]
            else:
                try:
                    directory = self.list_harvest_egresses(task_type)
                except (BrowsingAPIError, ConfigurationError) as e:
                    self.logger.warning(
                        "Live harvest egress check skipped: %s. The task has NOT "
                        "been blocked, and the check will not be retried for "
                        "%d seconds.",
                        e,
                        int(_DIRECTORY_TTL_SECONDS),
                    )
                    directory = None
                self._directory_cache[key] = (now, directory)
        if directory is None:
            return None
        if not directory:
            self.logger.info(
                "Live harvest egress check: the directory lists nothing in the "
                "%s pool for org %r, so there is nothing to check against. An "
                "org lookup sees only egress provisioned to that org and the "
                "orgs above it, so an org whose egress is provisioned to its "
                "sub-orgs lists nothing: set HARVEST_EGRESS_ORG to the sub-org "
                "the harvest token belongs to. Not checking.",
                key[0],
                org_name,
            )
            return None
        gap = describe_harvest_egress_gap(egress_info, directory)
        if gap and warn:
            self.logger.warning(
                "Live harvest egress check: %s. The task has NOT been blocked: "
                "harvest can use egress an org's Launcher Options hide from "
                "this list.",
                gap.rstrip("."),
            )
        return gap

    @api_tag(MethodType.CONVENIENCE, wraps=["find_harvest_task"])
    def wait_for_completion(
        self,
        task_id: str,
        max_wait_time: int = 1800,  # 30 minutes
        check_interval: int = 10,
        progress_callback: Callable[..., Any] | None = None,
    ) -> dict[str, Any]:
        """Wait for a harvest task to complete, with optional progress monitoring.

        Monitors a harvest task until it completes successfully, encounters an error,
        or the maximum wait time is exceeded. Provides optional progress callbacks
        for real-time monitoring.

        Args:
            task_id: The ID of the task to monitor
            max_wait_time: Maximum time to wait in seconds (default: 30 minutes).
                A deadline: the last status check happens at it, not one
                ``check_interval`` past it.
            check_interval: Time between status checks in seconds (default: 10)
            progress_callback: Optional function to call with progress updates

        Returns:
            The final task record, exactly as ``find_harvest_task()``
            returned it on the last poll — this method hands that
            dictionary straight back and neither renames nor adds any
            field. See ``find_harvest_task()`` for the full field list;
            in particular the timestamps are ``create_ts``/``start_ts``/
            ``end_ts``, and the ``created_ts``/``started_ts``/
            ``completed_ts`` aliases are **not** present here. Only
            ``wait_for_completion_async()`` adds those.

        Raises:
            HarvesterAPIError: If the task fails or times out. A task
                that reaches ``error`` status is raised, not returned,
                with the record's ``last_error`` in the message.
            ValidationError: If parameters are invalid

        Examples:
            >>> def progress_handler(task_info):
            ...     # No percentage is available; status is the signal.
            ...     print(f"Status: {task_info['status']}")
            >>>
            >>> result = api.wait_for_completion(
            ...     task_id="task_123",
            ...     max_wait_time=600,  # 10 minutes
            ...     progress_callback=progress_handler
            ... )
        """
        self._require_task_id(task_id)

        if not isinstance(max_wait_time, int) or max_wait_time <= 0:
            raise ValidationError("max_wait_time must be a positive integer")

        if not isinstance(check_interval, int) or check_interval <= 0:
            raise ValidationError("check_interval must be a positive integer")

        start_time = time.time()
        checks_performed = 0

        self.logger.info(
            "Waiting for task completion: %s (max %d seconds)", task_id, max_wait_time
        )

        try:
            while True:
                # Check if we've exceeded the maximum wait time
                elapsed_time = time.time() - start_time
                if elapsed_time > max_wait_time:
                    raise HarvesterAPIError(
                        f"Task {task_id} did not complete within {max_wait_time} seconds"
                    )

                # Get current task status
                try:
                    task_status = self.find_harvest_task(task_id)
                    if not isinstance(task_status, dict):
                        raise HarvesterAPIError(
                            f"Invalid task status format: {type(task_status)}"
                        )

                    status = task_status.get("status")
                    checks_performed += 1

                    # Call progress callback if provided
                    if progress_callback is not None:
                        try:
                            progress_callback(task_status)
                        except Exception as e:
                            self.logger.warning("Progress callback failed: %s", e)

                    # Check terminal states
                    if status == "done":
                        self.logger.info(
                            "Task %s completed successfully after %d checks (%.1f seconds)",
                            task_id,
                            checks_performed,
                            elapsed_time,
                        )
                        return task_status

                    elif status == "error":
                        error_msg = task_status.get("last_error", "Unknown error")
                        raise HarvesterAPIError(
                            f"Task {task_id} failed with server error: {error_msg}"
                        )

                except HarvesterAPIError as e:
                    # Re-check the original error message format
                    error_str = str(e)
                    if (
                        "task produced no output" in error_str
                        and "TypeError" in error_str
                    ):
                        # This indicates a response parsing issue from find_harvest_task
                        self.logger.error(
                            "Task status retrieval failed due to response format issue: %s",
                            e,
                        )
                        # Don't re-wrap the error message - just re-raise the original
                        raise
                    else:
                        raise

                if status in IN_PROGRESS_TASK_STATUSES:
                    # Task is still in progress
                    self.logger.debug(
                        "Task %s status: %s (check %d, %.1f seconds elapsed)",
                        task_id,
                        status,
                        checks_performed,
                        elapsed_time,
                    )
                    delay = _poll_delay(check_interval, max_wait_time, elapsed_time)
                    if delay is None:
                        raise HarvesterAPIError(
                            f"Task {task_id} did not complete within "
                            f"{max_wait_time} seconds"
                        )
                    time.sleep(delay)

                elif status not in ["done", "error"]:
                    raise HarvesterAPIError(f"Unexpected task status: {status}")

        except KeyboardInterrupt as exc:
            self.logger.info("Task monitoring interrupted by user")
            raise HarvesterAPIError("Task monitoring was interrupted") from exc

    @api_tag(MethodType.CONVENIENCE, wraps=["find_harvest_task"])
    def wait_for_completion_with_progress(
        self,
        task_id: str,
        task_type: str,
        max_wait_time: int = 1800,  # 30 minutes
        check_interval: int = 10,
        progress_callback: Callable[..., Any] | None = None,
    ) -> TaskProgress:
        """Wait for a harvest task to complete with enhanced progress tracking.

        Monitors a harvest task with detailed progress metrics including timing,
        status transitions, and estimated completion times.

        Args:
            task_id: The ID of the task to monitor
            task_type: Type of harvest task
            max_wait_time: Maximum time to wait in seconds (default: 30 minutes).
                A deadline: the last status check happens at it, not one
                ``check_interval`` past it.
            check_interval: Time between status checks in seconds (default: 10)
            progress_callback: Optional function to call with TaskProgress updates

        Returns:
            TaskProgress object with detailed metrics when completed

        Raises:
            HarvesterAPIError: If task fails or times out
            ValidationError: If parameters are invalid

        Examples:
            >>> def progress_handler(progress: TaskProgress):
            ...     print(f"Status: {progress.status} ({progress.progress_percentage}%)")
            ...     if progress.estimated_completion:
            ...         eta = datetime.fromtimestamp(progress.estimated_completion)
            ...         print(f"ETA: {eta}")
            >>>
            >>> progress = api.wait_for_completion_with_progress(
            ...     task_id="task_123",
            ...     task_type="asset",
            ...     progress_callback=progress_handler
            ... )
            >>> print(f"Completed in {progress.get_duration():.1f} seconds")
        """
        self._require_task_id(task_id)

        if not isinstance(max_wait_time, int) or max_wait_time <= 0:
            raise ValidationError("max_wait_time must be a positive integer")

        if not isinstance(check_interval, int) or check_interval <= 0:
            raise ValidationError("check_interval must be a positive integer")

        # Initialize progress tracker
        progress = TaskProgress(task_id, task_type)

        start_time = time.time()
        checks_performed = 0

        self.logger.info(
            "Waiting for task completion with progress tracking: %s (max %d seconds)",
            task_id,
            max_wait_time,
        )

        try:
            while True:
                # Check if we've exceeded the maximum wait time
                elapsed_time = time.time() - start_time
                if elapsed_time > max_wait_time:
                    progress.set_error(f"Task timeout after {max_wait_time} seconds")
                    raise HarvesterAPIError(
                        f"Task {task_id} did not complete within {max_wait_time} seconds"
                    )

                # Get current task status
                task_status = self.find_harvest_task(task_id)
                status: str = task_status.get("status", "unknown")
                checks_performed += 1

                # Update progress tracker
                # No progress argument: the platform reports state through
                # status and finished and sends no percentage, so this was
                # always None -- and it was being passed positionally into
                # update_status' "message" parameter, not its "progress" one.
                # Two mistakes that cancelled out only because the value never
                # existed.
                progress.update_status(status)
                progress.retry_count = checks_performed - 1

                # Call progress callback if provided
                if progress_callback is not None:
                    try:
                        progress_callback(progress)
                    except Exception as e:
                        self.logger.warning("Progress callback failed: %s", e)

                # Check terminal states
                if status == "done":
                    self.logger.info(
                        "Task %s completed successfully after %d checks (%.1f seconds)",
                        task_id,
                        checks_performed,
                        elapsed_time,
                    )
                    return progress

                elif status == "error":
                    error_msg = task_status.get("last_error", "Unknown error")
                    progress.set_error(error_msg)
                    raise HarvesterAPIError(
                        f"Task {task_id} failed with server error: {error_msg}"
                    )

                elif status in IN_PROGRESS_TASK_STATUSES:
                    # Task is still in progress
                    self.logger.debug(
                        "Task %s status: %s (check %d, %.1f seconds elapsed)",
                        task_id,
                        status,
                        checks_performed,
                        elapsed_time,
                    )
                    delay = _poll_delay(check_interval, max_wait_time, elapsed_time)
                    if delay is None:
                        progress.set_error(
                            f"Task timeout after {max_wait_time} seconds"
                        )
                        raise HarvesterAPIError(
                            f"Task {task_id} did not complete within "
                            f"{max_wait_time} seconds"
                        )
                    time.sleep(delay)

                else:
                    progress.set_error(f"Unexpected task status: {status}")
                    raise HarvesterAPIError(f"Unexpected task status: {status}")

        except KeyboardInterrupt as exc:
            progress.set_error("Task monitoring interrupted by user")
            self.logger.info("Task monitoring interrupted by user")
            raise HarvesterAPIError("Task monitoring was interrupted") from exc

    @api_tag(MethodType.CONVENIENCE, wraps=["find_harvest_task", "getfile"])
    def download_task_result(
        self,
        task_id: str,
        output_path: str,
        verify_completion: bool = True,
    ) -> str:
        """Download the result of a completed harvest task.

        Downloads the harvested content from a completed task to the local filesystem.
        Optionally verifies that the task is completed before attempting download.

        Args:
            task_id: The ID of the completed task
            output_path: Local path where to save the downloaded file
            verify_completion: Whether to verify task completion before download

        Returns:
            Path of the downloaded file

        Raises:
            HarvesterAPIError: If download fails or task is not completed
            ValidationError: If parameters are invalid

        Examples:
            >>> downloaded_file = api.download_task_result(
            ...     task_id="task_123",
            ...     output_path="/local/downloads/harvest_result.zip"
            ... )
            >>> print(f"Downloaded to: {downloaded_file}")
        """
        self._require_task_id(task_id)

        if not output_path:
            raise ValidationError("output_path must be provided")

        # Verify task completion if requested
        if verify_completion:
            task_status = self.find_harvest_task(task_id)
            if task_status.get("status") != "done":
                raise HarvesterAPIError(
                    f"Task {task_id} is not completed. "
                    f"Current status: {task_status.get('status')}"
                )

            result_file_id = task_status.get("result_file_id")
            if not result_file_id:
                raise HarvesterAPIError(f"No result file found for task {task_id}")
        else:
            # Get task details to find result file ID
            task_status = self.find_harvest_task(task_id)
            result_file_id = task_status.get("result_file_id")
            if not result_file_id:
                raise HarvesterAPIError(f"No result file found for task {task_id}")

        try:
            self.logger.debug("Downloading result for task: %s", task_id)
            downloaded_path = self.file_api.download_file(result_file_id, output_path)

            self.logger.info(
                "Successfully downloaded task result: %s -> %s",
                task_id,
                downloaded_path,
            )
            return downloaded_path

        except FileAPIError as e:
            raise HarvesterAPIError(f"Failed to download task result: {e}") from e

    @api_tag(MethodType.UTILITY)
    @classmethod
    def get_valid_params(cls, task_type: str) -> set[str]:
        """Get valid parameters for a specific task type.

        Args:
            task_type: Harvest task type; see :attr:`VALID_TASK_TYPES`

        Returns:
            Set of valid parameter names for the task type

        Raises:
            ValidationError: If task_type is invalid

        Examples:
            >>> asset_params = HarvesterAPI.get_valid_params("asset")
            >>> print("Valid asset parameters:")
            >>> for param in sorted(asset_params):
            ...     print(f"  - {param}")
        """
        if task_type in WGET_PARAMS_TASK_TYPES:
            return cls.VALID_ASSET_PARAMS.copy()
        elif task_type == "visual":
            return cls.VALID_VISUAL_PARAMS.copy()
        elif task_type == "video":
            return cls.VALID_VIDEO_PARAMS.copy()
        else:
            raise ValidationError(
                f"Invalid task_type: {task_type}. "
                f"Valid types are: {', '.join(sorted(cls.VALID_TASK_TYPES))}"
            )

    @api_tag(MethodType.UTILITY)
    @classmethod
    def get_valid_task_types(cls) -> set[str]:
        """Get all valid task types.

        Returns:
            Set of valid task type names

        Examples:
            >>> task_types = HarvesterAPI.get_valid_task_types()
            >>> print("Supported task types:")
            >>> for task_type in sorted(task_types):
            ...     print(f"  - {task_type}")
        """
        return set(cls.VALID_TASK_TYPES)

    # Async Methods (require httpx)

    @api_tag(
        MethodType.ASYNC_VARIANT,
        api_command="create_harvest_task",
        sync_of="create_harvest_task",
    )
    async def create_harvest_task_async(
        self,
        task_type: str,
        urls: list[str],
        egress_info: dict[str, Any],
        dest_path: str = "/",
        dest_name: str | None = None,
        dest_bucket_id: str | None = None,
        task_params: dict[str, Any] | None = None,
        max_retries: int = 5,
        backoff_factor: int = 2,
        run_after: int | float | str | None = None,
        validate_egress: bool | None = None,
    ) -> str:
        """Asynchronously create a new harvest task for web content collection.

        This async version allows for non-blocking task creation and can be used
        in concurrent workflows for creating multiple tasks simultaneously.

        Args:
            task_type: Type of harvest task ("asset", "single", "visual",
                "video", or "wget")
            urls: Array containing exactly one URL string to harvest
            egress_info: Egress location information containing:
                - name: Egress location name (required, e.g., "New York, NY").
                  "asset" and "wget" tasks draw from the six asset datacenter
                  locations (Singapore, Dubai, Frankfurt, Sao Paulo,
                  Johannesburg, "New York, NY"): pass one of those or
                  "World". Any other name logs a warning rather than raising.
                - categories: Optional object with connectivity, availability, protocol
                  Each category field can be a string value or null. For
                  non-"asset" task types, connectivity/protocol are validated
                  against what is actually available at name.
            dest_path: Destination path in storage (defaults to root "/")
            dest_name: Name for the result file (auto-generated if not provided)
            dest_bucket_id: Destination bucket ID (uses default if not provided)
            task_params: Dictionary of task-specific parameters
            max_retries: Maximum number of retries for explicit HTTP 429
                rate-limit responses (default: 5). Ambiguous transport failures
                are not retried because task creation is non-idempotent.
            backoff_factor: Positive integer exponential backoff multiplier for
                rate-limit retries (default: 2)
            run_after: Optional schedule time — Unix epoch (int or float) or
                timezone-aware ISO 8601 string (e.g., "2026-02-13T18:30:00Z"
                or "2026-02-13T13:30:00-05:00"). Booleans and timezone-less
                strings are refused. When provided the task
                enters "wait" status until the specified time.
            validate_egress: Look the egress up in its pool's live directory
                (:meth:`check_harvest_egress`) and log a warning if it is not
                listed. Never blocks the task. ``None`` (default) uses the
                ``LIVE_EGRESS_VALIDATION`` config key.

        Returns:
            Task ID string for monitoring and managing the harvest task

        Raises:
            HarvesterAPIError: If task creation fails
            ValidationError: If parameters are invalid

        Examples:
            >>> import asyncio
            >>> async def create_tasks():
            ...     harvester = HarvesterAPI(config)
            ...     task_id = await harvester.create_harvest_task_async(
            ...         task_type="asset",
            ...         urls=["https://example.com/files/"],
            ...         egress_info={"name": "New York, NY"},
            ...         dest_path="/downloads/"
            ...     )
            ...     return task_id
            >>> task_id = asyncio.run(create_tasks())
        """
        validate_positive_int(max_retries, "max_retries")
        validate_positive_int(backoff_factor, "backoff_factor")

        task_params = self._normalize_task_params(task_params)
        top_level_keys = self._top_level_keys_for_type(task_type)
        validate_task_creation_params(
            task_type,
            urls,
            egress_info,
            dest_path,
            task_params,
            self.VALID_TASK_TYPES,
            self._valid_params_map,
            top_level_keys=top_level_keys,
        )

        if run_after is not None:
            validate_run_after(run_after)
        if self.live_egress_validation if validate_egress is None else validate_egress:
            # The lookup is a blocking request; keep it off the event loop.
            await asyncio.to_thread(self.check_harvest_egress, task_type, egress_info)

        normalized_egress_info = normalize_egress_info(egress_info)
        bucket_id = dest_bucket_id or self.dest_bucket_id
        final_dest_name = generate_dest_name_if_needed(dest_name, task_type)

        payload_commands = build_task_creation_payload(
            task_type,
            urls,
            normalized_egress_info,
            bucket_id,
            dest_path,
            final_dest_name,
            task_params or {},
            self.file_api.auth_token,
            run_after=run_after,
            top_level_keys=top_level_keys,
        )

        # Build API payload
        api_url = f"{self.base_url.rstrip('/')}/api/"
        payload = [{"command": "setauth", "data": self.auth_token}] + payload_commands

        async with httpx.AsyncClient(verify=self._ssl_context) as client:
            for attempt in range(max_retries):
                try:
                    response = await client.post(
                        api_url,
                        json=payload,
                        headers={
                            "Accept": "application/json",
                            "Content-Type": "application/json",
                        },
                        timeout=httpx.Timeout(30),
                    )
                    if response.status_code == 200:
                        task_response = response.json()
                        # A rejected command still returns HTTP 200, with the
                        # error in place of (or nested inside) the result.
                        # Without this the server's message was lost behind
                        # "Unexpected response format: [...]".
                        try:
                            result_inner = self._extract_api_result(task_response)
                        except SiloError as e:
                            raise HarvesterAPIError(
                                f"Failed to create async harvest task: {e.message}",
                                response_data=e.response_data,
                            ) from e
                        result_task_id: str | None = (
                            result_inner.get("task_id")
                            if isinstance(result_inner, dict)
                            else None
                        )
                        if result_task_id:
                            self.logger.info(
                                "Created async harvest task: %s", result_task_id
                            )
                            return result_task_id

                        raise HarvesterAPIError(
                            "Unexpected response format: "
                            f"{self._format_response_for_log(task_response)}"
                        )

                    elif response.status_code == 429:
                        if attempt == max_retries - 1:
                            break
                        try:
                            retry_after = int(
                                response.headers.get(
                                    "Retry-After", backoff_factor**attempt
                                )
                            )
                        except ValueError:
                            retry_after = int(backoff_factor**attempt)
                        self.logger.warning(
                            "Rate limit hit, retrying after %s seconds...", retry_after
                        )
                        await asyncio.sleep(retry_after)
                    else:
                        raise HarvesterAPIError(
                            f"API request failed with status code {response.status_code}"
                        )

                except httpx.HTTPError as e:
                    raise HarvesterAPIError(
                        "Harvest task creation failed after the request may have "
                        "reached the server; the task may have been created. "
                        "The request was not retried to avoid creating a duplicate: "
                        f"{e}"
                    ) from e
                except HarvesterAPIError:
                    raise
                except Exception as e:
                    raise HarvesterAPIError(
                        f"Failed to create async harvest task: {e}"
                    ) from e

        raise HarvesterAPIError("Rate-limit retries exhausted; task was not created")

    @api_tag(
        MethodType.ASYNC_VARIANT,
        api_command="find_harvest_task",
        sync_of="wait_for_completion",
    )
    async def wait_for_completion_async(
        self,
        task_id: str,
        max_wait_time: int = 1800,  # 30 minutes
        check_interval: int = 10,
        progress_callback: Callable[..., Any] | None = None,
    ) -> dict[str, Any]:
        """Asynchronously wait for a harvest task to complete.

        This async version allows for non-blocking task monitoring and can be used
        to monitor multiple tasks concurrently.

        Args:
            task_id: The ID of the task to monitor
            max_wait_time: Maximum time to wait in seconds (default: 30 minutes).
                A deadline: the last status check happens at it, not one
                ``check_interval`` past it.
            check_interval: Time between status checks in seconds (default: 10)
            progress_callback: Optional function to call with progress updates

        Returns:
            Final task status dictionary when completed. Shape matches the
            synchronous ``wait_for_completion()``: the raw server task dict
            merged with the normalized fields ``parse_task_response()``
            produces. No server-reported field is dropped.

            Server-reported, passed through unchanged. These carry the same
            meanings as in ``find_harvest_task()``, which documents each one
            in full:

            - task_id: Task identifier
            - status: One of ``wait``, ``claimed``, ``done`` or ``error``.
                Always ``done`` on a successful return here, since an ``error``
                status raises instead
            - finished: True once the task has been resolved
            - attempt_count: Attempts made, counting the last one
            - last_error: Failure message, null on the success path
            - hostname: Worker that ran the task
            - org_id: Organization that owns the task
            - create_ts: Creation time, a Unix epoch float
            - start_ts: Claim time of the most recent attempt
            - end_ts: Resolution time
            - run_after: Earliest time the task was allowed to be claimed
            - request_type: Task type, stripped and lower-cased
            - target: Worker pool the task was routed to
            - name: Caller-supplied label
            - task_params: Parameters the task was created with
            - dest_bucket_id: Storage bucket holding the results
            - dest_path: Path within that bucket
            - dest_name: Name of the result object
            - egress_info: Egress selection in effect
            - egress_tag: Superseded egress selector, null in practice
            - result_file_id: File ID of the result
            - task_results: Raw result payload from the worker

            Added on top by ``parse_task_response()``. These are aliases, so
            each one duplicates a field above under a different name rather
            than replacing it:

            - created_ts: Alias of ``create_ts``
            - started_ts: Alias of ``start_ts``
            - completed_ts: Alias of ``end_ts``
            - task_metadata: ``task_results["task-metadata"]``, or an empty
                dict when the worker recorded none

            Note:
                ``find_harvest_task()`` returns the raw record **without**
                these four aliases. Code that polls with one and finishes
                with the other must read ``create_ts`` from the poll result
                and can read either name from this one.

        Raises:
            HarvesterAPIError: If task fails or times out
            ValidationError: If parameters are invalid

        Examples:
            >>> import asyncio
            >>> async def monitor_task():
            ...     harvester = HarvesterAPI(config)
            ...     def progress_handler(task_info):
            ...         print(f"Status: {task_info['status']}")
            ...
            ...     result = await harvester.wait_for_completion_async(
            ...         task_id="task_123",
            ...         progress_callback=progress_handler
            ...     )
            ...     return result
            >>> result = asyncio.run(monitor_task())
        """
        self._require_task_id(task_id)

        if not isinstance(max_wait_time, int) or max_wait_time <= 0:
            raise ValidationError("max_wait_time must be a positive integer")

        if not isinstance(check_interval, int) or check_interval <= 0:
            raise ValidationError("check_interval must be a positive integer")

        # At least one check: an interval longer than the wait used to mean
        # none, so the call timed out without ever asking for the status.
        max_retries = max(1, max_wait_time // check_interval)
        api_url = f"{self.base_url.rstrip('/')}/api/"

        self.logger.info("Starting async status check loop for task ID: %s", task_id)
        retries = 0
        last_response = None
        start_time = time.time()

        try:
            async with httpx.AsyncClient(verify=self._ssl_context) as client:
                while retries < max_retries:
                    # Bound the loop by wall-clock elapsed time in addition to
                    # retry count. Retry count alone assumes each iteration
                    # costs ~check_interval seconds, but a slow API response
                    # (up to the 30s per-request timeout below) isn't counted
                    # against max_wait_time, which lets the real wait exceed
                    # it. This check only ever exits *earlier* than the retry
                    # count would (never later), so it doesn't change
                    # behavior when checks are fast.
                    elapsed_time = time.time() - start_time
                    if elapsed_time > max_wait_time:
                        self.logger.warning(
                            "Wall-clock max_wait_time (%s seconds) exceeded for "
                            "task %s before the retry budget was exhausted.",
                            max_wait_time,
                            task_id,
                        )
                        break

                    self.logger.debug(
                        f"Checking status of task {task_id} "
                        f"(attempt {retries + 1}/{max_retries})"
                    )

                    payload = [
                        {"command": "setauth", "data": self.auth_token},
                        {"command": "find_harvest_task", "task_id": task_id},
                    ]

                    try:
                        response = await client.post(
                            api_url,
                            json=payload,
                            headers={
                                "Accept": "application/json",
                                "Content-Type": "application/json",
                            },
                            timeout=httpx.Timeout(30),
                        )
                        response.raise_for_status()
                        last_response = response.json()

                        # Both error shapes ({"error": ...} and a nested
                        # {"result": {"error": ...}}) raise here, as the sync
                        # find_harvest_task does. Reading result directly
                        # turned the first into [] and polled until timeout,
                        # and hit task_result[0] on the second (KeyError).
                        try:
                            task_result = self._extract_api_result(last_response)
                        except SiloError as e:
                            if e.response_data is None:
                                self.logger.warning(
                                    "Invalid response format for task %s", task_id
                                )
                                raise HarvesterAPIError(
                                    f"Invalid response format for task {task_id}"
                                ) from e
                            raise HarvesterAPIError(
                                f"Failed to find harvest task {task_id}: {e.message}",
                                response_data=e.response_data,
                            ) from e
                        if task_result and not (
                            isinstance(task_result, list)
                            and isinstance(task_result[0], dict)
                        ):
                            raise HarvesterAPIError(
                                f"Unexpected task result for task {task_id}: "
                                f"{self._format_response_for_log(last_response)}"
                            )

                        if task_result:
                            task_status = task_result[0].get("status", "")
                            self.logger.info(f"Task {task_id} status: {task_status}")

                            # Call progress callback if provided
                            if progress_callback is not None:
                                try:
                                    progress_callback(task_result[0])
                                except Exception as e:
                                    self.logger.warning(
                                        "Progress callback failed: %s", e
                                    )

                            if task_status == "done":
                                self.logger.info(
                                    f"Task {task_id} completed successfully."
                                )
                                task_info = parse_task_response(last_response)
                                if not task_info:
                                    raise HarvesterAPIError(
                                        f"Failed to parse task response for {task_id}"
                                    )
                                # Align this method's return shape with the
                                # synchronous wait_for_completion(), which
                                # returns the full raw server task dict (see
                                # find_harvest_task/task_result[0] above).
                                # parse_task_response() only surfaces 9
                                # hardcoded keys and drops the rest, including
                                # task_params, attempt_count, hostname,
                                # finished, target and task_results — merge the
                                # raw dict as the base and layer the normalized
                                # fields on top so nothing the server sent is
                                # lost. (This list previously cited "progress"
                                # and "worker_id"; the platform sends neither,
                                # so the justification for the merge rested on
                                # two fields that do not exist.)
                                full_task_info: dict[str, Any] = {
                                    **task_result[0],
                                    **task_info,
                                }
                                return full_task_info
                            elif task_status == "error":
                                error_msg = task_result[0].get(
                                    "last_error", "Unknown error"
                                )
                                raise HarvesterAPIError(
                                    f"Task {task_id} failed with server error: {error_msg}"
                                )
                            elif task_status == "wait":
                                run_after = task_result[0].get("run_after")
                                if run_after:
                                    run_after_time = datetime.fromtimestamp(
                                        run_after, tz=UTC
                                    )
                                    self.logger.info(
                                        f"Task {task_id} is in 'wait' status, "
                                        f"scheduled to run at: {run_after_time} UTC."
                                    )
                                else:
                                    self.logger.info(
                                        f"Task {task_id} is in 'wait' status, "
                                        f"but no scheduled time is available."
                                    )
                            elif task_status in ACTIVE_TASK_STATUSES:
                                self.logger.info(
                                    f"Task {task_id} is in progress... "
                                    f"Status: {task_status}"
                                )
                            else:
                                self.logger.warning(
                                    f"Unexpected task status for task {task_id}: "
                                    f"{task_status}"
                                )
                        else:
                            self.logger.warning(
                                f"No task result found for task {task_id}"
                            )

                    except httpx.HTTPError as e:
                        self.logger.error(
                            f"Error: Request failed for task {task_id} with exception {e}."
                        )
                        raise HarvesterAPIError(
                            f"Request failed for task {task_id}: {e}"
                        ) from e

                    delay = _poll_delay(check_interval, max_wait_time, elapsed_time)
                    if delay is None:
                        break
                    await asyncio.sleep(delay)
                    retries += 1

            # Either the retry budget or the wall-clock max_wait_time was
            # exhausted without the task reaching a terminal state.
            self.logger.error(
                f"Timed out waiting for task {task_id}. "
                f"Task may not have completed successfully."
            )
            raise HarvesterAPIError(
                f"Task {task_id} did not complete within {max_wait_time} seconds"
            )

        except HarvesterAPIError:
            raise
        except Exception as e:
            raise HarvesterAPIError(f"Failed to monitor async task: {e}") from e

    @api_tag(
        MethodType.ASYNC_CONVENIENCE,
        wraps=[
            "create_harvest_task_async",
            "wait_for_completion_async",
            "download_file_chunked_async",
            "download_file_with_retry_async",
        ],
    )
    async def process_harvest_workflow_async(
        self,
        task_type: str,
        urls: list[str],
        egress_info: dict[str, Any],
        output_dir: str,
        dest_path: str = "/",
        dest_name: str | None = None,
        task_params: dict[str, Any] | None = None,
        use_chunked_download: bool = False,
        max_wait_time: int = 1800,
        check_interval: int = 20,
    ) -> dict[str, Any] | None:
        """Process a complete harvest workflow asynchronously.

        This high-level async method orchestrates the entire harvest workflow:
        1. Creates a harvest task
        2. Monitors its status until completion
        3. Downloads the result file
        4. Returns comprehensive task information

        Args:
            task_type: Type of harvest task ("asset", "single", "visual",
                "video", or "wget")
            urls: Array containing exactly one URL string to harvest
            egress_info: Egress location information containing:
                - name: Egress location name (required, e.g., "New York, NY").
                  "asset" and "wget" tasks draw from the six asset datacenter
                  locations (Singapore, Dubai, Frankfurt, Sao Paulo,
                  Johannesburg, "New York, NY"): pass one of those or
                  "World". Any other name logs a warning rather than raising.
                - categories: Optional object with connectivity, availability, protocol
                  Each category field can be a string value or null. For
                  non-"asset" task types, connectivity/protocol are validated
                  against what is actually available at name.
            output_dir: Directory to save downloaded results
            dest_path: Destination path in storage (default: "/")
            dest_name: Name for result file (auto-generated if not provided)
            task_params: Dictionary of task-specific parameters
            use_chunked_download: Whether to use chunked download (default: False)
            max_wait_time: Maximum time to wait for completion (default: 30 minutes)
            check_interval: Time between status checks (default: 20 seconds)

        Returns:
            Dictionary with task information and results, None if failed

        Raises:
            HarvesterAPIError: If workflow fails
            ValidationError: If parameters are invalid

        Examples:
            >>> import asyncio
            >>> async def harvest_site():
            ...     harvester = HarvesterAPI(config)
            ...     result = await harvester.process_harvest_workflow_async(
            ...         task_type="asset",
            ...         urls=["https://example.com/documents/"],
            ...         egress_info={"name": "New York, NY"},
            ...         output_dir="./downloads",
            ...         task_params={"recursive": True, "level": 2}
            ...     )
            ...     return result
            >>> result = asyncio.run(harvest_site())
        """
        # Validate egress_info with task-type-aware validation
        validate_egress_for_task_type(task_type, egress_info)

        # Create output directory
        output_path = Path(output_dir)
        await asyncio.to_thread(output_path.mkdir, parents=True, exist_ok=True)

        self.logger.info(
            "Processing %s task for URL: %s", task_type, urls[0] if urls else "<empty>"
        )

        try:
            # Step 1: Create harvest task
            task_id = await self.create_harvest_task_async(
                task_type=task_type,
                urls=urls,
                egress_info=egress_info,
                dest_path=dest_path,
                dest_name=dest_name,
                task_params=task_params,
            )

            self.logger.info("Started %s task with ID: %s", task_type, task_id)

            # Step 2: Monitor task status
            task_info = await self.wait_for_completion_async(
                task_id=task_id,
                max_wait_time=max_wait_time,
                check_interval=check_interval,
            )

            if not task_info:
                self.logger.error("No status response received for task %s", task_id)
                return None

            result_file_id = task_info.get("result_file_id")
            if not result_file_id:
                self.logger.error("Missing result file ID for task %s", task_id)
                return task_info

            # Step 3: Download result file
            output_file_path = output_path / f"harvest-{task_id}.zip"

            try:
                self.logger.info("Downloading result for task %s", task_id)

                if use_chunked_download:
                    downloaded_file = await self.file_api.download_file_chunked_async(
                        file_id=result_file_id, output_path=str(output_file_path)
                    )
                else:
                    downloaded_file = (
                        await self.file_api.download_file_with_retry_async(
                            file_id=result_file_id, output_path=str(output_file_path)
                        )
                    )

                if downloaded_file:
                    self.logger.info("Downloaded file saved to: %s", downloaded_file)
                    task_info["downloaded_file"] = str(downloaded_file)
                    task_info["download_success"] = True
                else:
                    self.logger.error(
                        "Failed to download result file for task %s", task_id
                    )
                    task_info["download_success"] = False

            except Exception as e:
                self.logger.error("Error downloading file for task %s: %s", task_id, e)
                task_info["download_success"] = False
                task_info["download_error"] = str(e)

            return task_info

        except Exception as e:
            if isinstance(e, (HarvesterAPIError, ValidationError)):
                raise
            raise HarvesterAPIError(
                f"Failed to process async harvest workflow: {e}"
            ) from e

    @api_tag(
        MethodType.ASYNC_CONVENIENCE,
        wraps=["process_harvest_workflow_async"],
    )
    async def bulk_create_and_monitor_async(
        self,
        tasks: list[dict[str, Any]],
        output_dir: str,
        max_concurrent: int = 5,
        egress_info: dict[str, Any] | None = None,
    ) -> list[dict[str, Any] | None]:
        """Create and monitor multiple harvest tasks concurrently.
        This method processes multiple harvest tasks in parallel with configurable
        concurrency limits to efficiently handle bulk harvesting operations.

        Args:
            tasks: List of task dictionaries, each containing:
                - task_type: Type of harvest task
                - urls: List containing exactly one URL string to harvest
                  (canonical key — matches ``create_harvest_task()``).
                  A single ``url`` string key is also accepted as a
                  convenience alias and is wrapped into a one-item list.
                - egress_info: Optional per-task egress location dict. When
                  provided, it overrides the bulk-level ``egress_info`` for
                  this task only — this is how you mix asset tasks (which
                  need one of the 6 datacenter locations) with non-asset
                  tasks in the same call.
                - params: Task-specific parameters (dict format)
                - dest_path: Optional destination path (default: "/")
                - dest_name: Optional destination name
            output_dir: Directory to save downloaded results
            max_concurrent: Positive maximum number of concurrent tasks (default: 5)
            egress_info: Default egress location information used for any
                task that doesn't specify its own ``egress_info``. When
                omitted, each task gets the default for its type:
                ``{"name": "New York, NY"}`` for ``"asset"`` and ``"wget"``,
                ``{"name": "World"}`` (any location) for every other type.
                Note: asset and wget tasks draw from the six asset
                datacenter locations (Singapore, Dubai, Frankfurt, Sao Paulo,
                Johannesburg, ``"New York, NY"``) or ``"World"``. Another name,
                such as ``"London"``, only logs a warning, and the task fails
                at run time unless an asset location sits beneath it. If you
                pass a bulk-level value and mix asset and non-asset tasks, set
                ``egress_info`` per-task (see above) or use separate calls.

        Returns:
            List of task results (same order as input), None for failed tasks

        Raises:
            HarvesterAPIError: If bulk processing fails
            ValidationError: If parameters are invalid

        Examples:
            >>> import asyncio
            >>> async def bulk_harvest():
            ...     harvester = HarvesterAPI(config)
            ...     tasks = [
            ...         {
            ...             "task_type": "asset",
            ...             "urls": ["https://site1.com"],
            ...             "egress_info": {"name": "New York, NY"},
            ...             "params": {"recursive": True}
            ...         },
            ...         {
            ...             "task_type": "visual",
            ...             "urls": ["https://site2.com"],
            ...             "params": {"output_pdf": True}
            ...         }
            ...     ]
            ...     results = await harvester.bulk_create_and_monitor_async(
            ...         tasks=tasks,
            ...         output_dir="./downloads",
            ...         max_concurrent=3
            ...     )
            ...     return results
            >>> results = asyncio.run(bulk_harvest())
        """
        validate_positive_int(max_concurrent, "max_concurrent")

        # Create semaphore to limit concurrent tasks
        semaphore = asyncio.Semaphore(max_concurrent)

        async def process_single_task(
            task_config: dict[str, Any],
        ) -> dict[str, Any] | None:
            """Process a single task with semaphore control."""
            async with semaphore:
                try:
                    # Get task_params from either 'params' or 'task_params' key
                    task_params = task_config.get("task_params") or task_config.get(
                        "params", {}
                    )

                    # Accept both 'urls' (list — canonical, matches
                    # create_harvest_task/create_harvest_task_async) and a
                    # single 'url' string as a convenience alias, so callers
                    # don't hit a KeyError from a docstring/param mismatch.
                    task_urls = task_config.get("urls")
                    if task_urls is None:
                        single_url = task_config.get("url")
                        task_urls = [single_url] if single_url is not None else None
                    if not task_urls:
                        raise ValidationError(
                            "Each task dict must include a non-empty 'urls' "
                            "list (or a single 'url' string)"
                        )

                    # Honor this task's own egress_info when provided,
                    # falling back to the bulk-level default. Previously the
                    # outer-scope egress_info was always used, making it
                    # impossible to mix asset tasks (which require one of
                    # the 6 datacenter locations) with non-asset tasks in a
                    # single bulk call, despite the docstring advertising
                    # per-task egress as the way to do so.
                    #
                    # With neither, use the default for this task's type: a
                    # single bulk-wide default sent "New York, NY" to asset
                    # tasks, which only accept the datacenter locations.
                    task_egress_info = task_config.get("egress_info", egress_info)
                    if task_egress_info is None:
                        task_egress_info = default_egress_for_task_type(
                            task_config["task_type"]
                        )

                    return await self.process_harvest_workflow_async(
                        task_type=task_config["task_type"],
                        urls=task_urls,
                        egress_info=task_egress_info,
                        output_dir=output_dir,
                        dest_path=task_config.get("dest_path", "/"),
                        dest_name=task_config.get("dest_name"),
                        task_params=(
                            task_params if isinstance(task_params, dict) else None
                        ),
                    )
                except Exception as e:
                    self.logger.error("Task failed: %s", e)
                    return None

        # Process all tasks concurrently
        self.logger.info(
            "Processing %d tasks with max concurrency: %d", len(tasks), max_concurrent
        )

        try:
            results = await asyncio.gather(
                *[process_single_task(task) for task in tasks]
            )

            processed_results: list[dict[str, Any] | None] = [
                r if isinstance(r, dict) else None for r in results
            ]

            successful_tasks = sum(1 for r in processed_results if r is not None)
            self.logger.info(
                f"Completed bulk processing: {successful_tasks}/{len(tasks)} tasks successful"
            )

            return processed_results

        except Exception as e:
            if isinstance(e, (HarvesterAPIError, ValidationError)):
                raise
            raise HarvesterAPIError(f"Failed to process bulk async harvest: {e}") from e

__init__

__init__(config: dict[str, Any]) -> None

Initialize the harvester API client.

Parameters:

Name Type Description Default
config dict[str, Any]

Configuration dictionary containing API settings

required

Raises:

Type Description
ConfigurationError

If required configuration is missing or invalid

Source code in silo_sdk/harvesting/harvester_api.py
def __init__(self, config: dict[str, Any]) -> None:
    """Initialize the harvester API client.

    Args:
        config: Configuration dictionary containing API settings

    Raises:
        ConfigurationError: If required configuration is missing or invalid
    """
    super().__init__(config)

    # Comprehensive configuration validation
    validation_errors = validate_harvester_config(config)
    if validation_errors:
        raise ConfigurationError(
            f"Harvester configuration validation failed: {'; '.join(validation_errors)}"
        )

    # Basic required configuration (legacy support)
    validate_config_for_api(config, ["SCRAPE_TOKEN", "BUCKET_ID"])

    self.auth_token = config["SCRAPE_TOKEN"]
    self.dest_bucket_id = config["BUCKET_ID"]

    # Off by default; one directory lookup per pool per minute when on.
    self.live_egress_validation = _config_flag(
        config.get("LIVE_EGRESS_VALIDATION", False)
    )
    self._browsing: Any = None
    # None caches a failed lookup, so a bulk run warns once, not per task.
    self._directory_cache: dict[
        tuple[str, str], tuple[float, list[EgressDirectoryEntry] | None]
    ] = {}
    # The async methods reach check_harvest_egress via asyncio.to_thread,
    # so the cache is filled from several threads. Two levels of lock:
    # _directory_lock guards _pool_locks and the lazy BrowsingAPI and is
    # only held briefly; each (pool, org) has its own lock, held across
    # that pool's lookup, under which its cache entry is read and written.
    # Concurrent checks of one pool share a single request, and a slow
    # lookup for one pool never stalls checks of another. Lock order is
    # always pool lock, then _directory_lock (inside _browsing_api()).
    self._directory_lock = threading.RLock()
    self._pool_locks: dict[tuple[str, str], threading.Lock] = {}

    # Initialize task status cache if enabled
    self._status_cache_enabled, self._status_cache_ttl, self._status_cache = (
        init_status_cache(config)
    )

    # Initialize file API for downloading results
    self.file_api = FileAPI(config)
    self._ssl_context = self.create_ssl_context()

    self.logger.info("Initialized HarvesterAPI client with validated configuration")

__enter__

__enter__() -> HarvesterAPI

Enter context manager for resource management.

Returns:

Type Description
HarvesterAPI

Self for use in with statement

Source code in silo_sdk/harvesting/harvester_api.py
def __enter__(self) -> HarvesterAPI:
    """Enter context manager for resource management.

    Returns:
        Self for use in with statement
    """
    self.logger.debug("Entering HarvesterAPI context manager")
    return self

__exit__

__exit__(exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: TracebackType | None) -> None

Exit context manager and perform cleanup.

Parameters:

Name Type Description Default
exc_type type[BaseException] | None

Exception type if an exception occurred

required
exc_val BaseException | None

Exception value if an exception occurred

required
exc_tb TracebackType | None

Exception traceback if an exception occurred

required
Source code in silo_sdk/harvesting/harvester_api.py
def __exit__(
    self,
    exc_type: type[BaseException] | None,
    exc_val: BaseException | None,
    exc_tb: TracebackType | None,
) -> None:
    """Exit context manager and perform cleanup.

    Args:
        exc_type: Exception type if an exception occurred
        exc_val: Exception value if an exception occurred
        exc_tb: Exception traceback if an exception occurred
    """
    # Suppress unused parameter warnings - these are required by the context manager protocol
    _ = exc_type, exc_val, exc_tb
    try:
        self.cleanup_resources()
    except Exception as e:
        self.logger.warning("Error during context manager cleanup: %s", e)
    finally:
        self.logger.debug("Exited HarvesterAPI context manager")

cleanup_resources

cleanup_resources() -> None

Clean up resources and perform housekeeping tasks.

This method should be called when done with the HarvesterAPI instance to properly clean up resources, clear caches, and close connections.

Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.UTILITY)
def cleanup_resources(self) -> None:
    """Clean up resources and perform housekeeping tasks.

    This method should be called when done with the HarvesterAPI instance
    to properly clean up resources, clear caches, and close connections.
    """
    try:
        # Clear status cache
        if self._status_cache:
            cache_size = len(self._status_cache)
            self._status_cache.clear()
            if cache_size > 0:
                self.logger.info("Cleared %d cached task statuses", cache_size)

        # Close every HTTP session this instance owns.
        self.close()

    except Exception as e:
        self.logger.error("Error during resource cleanup: %s", e)

close

close() -> None

Close every HTTP session this instance owns.

That is its own session, the session of the FileAPI it builds for downloads, and the BrowsingAPI the live egress check creates on first use. Safe to call more than once. A failure closing one does not leave the others open: all three are attempted, and an error is raised afterwards (if more than one fails, the last error is raised with the earlier one chained as its context).

Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.UTILITY)
def close(self) -> None:
    """Close every HTTP session this instance owns.

    That is its own session, the session of the FileAPI it builds for
    downloads, and the BrowsingAPI the live egress check creates on first
    use. Safe to call more than once. A failure closing one does not
    leave the others open: all three are attempted, and an error is
    raised afterwards (if more than one fails, the last error is raised
    with the earlier one chained as its context).
    """
    try:
        super().close()
    finally:
        try:
            file_api = getattr(self, "file_api", None)
            if file_api is not None:
                file_api.close()
        finally:
            self._close_browsing()

create_harvest_task

create_harvest_task(task_type: str, urls: list[str], egress_info: dict[str, Any], dest_path: str = '/', dest_name: str | None = None, dest_bucket_id: str | None = None, task_params: dict[str, Any] | None = None, run_after: int | float | str | None = None, validate_egress: bool | None = None) -> str

Create a new harvest task for web content collection.

Creates a harvesting task that will collect content from the specified URL using the given parameters and store the results in the designated location.

Parameters:

Name Type Description Default
task_type str

Type of harvest task. Must be one of "asset", "visual", "video", "single", or "wget".

  • "visual" — Full-page capture. Output format controlled by task_params:

  • Screenshot (default): no params, or {"output_image": True}

  • PDF: {"output_pdf": True}
  • MHTML: {"output_mhtml": True}

IMPORTANT: "screenshot", "pdf", and "mhtml" are not valid task_type values. Always use "visual" with task_params.

  • "asset" — Recursive asset download (HTML, JS, CSS, images). Control depth/filters via task_params (wget-style options).
  • "wget" — Accepted as its own type and run as an "asset" task, with the same parameters.
  • "video" — Video recording of page interaction (youtube-dl-style options).
required
urls list[str]

List containing exactly one URL string to harvest.

required
egress_info dict[str, Any]

Egress location dict with required "name" key (e.g., {"name": "New York, NY"}). "asset" and "wget" tasks draw from the six asset datacenter locations (Singapore, Dubai, Frankfurt, Sao Paulo, Johannesburg, "New York, NY"): pass one of those or "World". Any other name, such as "London", logs a warning rather than raising, since a region or country works when an asset location sits beneath it. Optional "categories" key accepts "connectivity", "availability", "protocol" (direct, tor or sovereign); omitted keys are not sent. A server_set key is removed before sending (the server rejects it for harvest tasks). For non-"asset" task types, "connectivity"/"protocol" are validated against what is actually available at "name" (see silo_sdk.harvesting.egress_validation). Does not apply to "asset" tasks. These checks are advisory. For the live answer, see :meth:list_harvest_egresses and validate_egress below: visual, video and single tasks draw from the same pool as browsing contexts, asset and wget tasks from their own.

required
dest_path str

Destination path in storage (defaults to "/")

'/'
dest_name str | None

Name for the result file (auto-generated if not provided)

None
dest_bucket_id str | None

Destination bucket ID (uses default if not provided)

None
task_params dict[str, Any] | None

Task-specific output parameters. Most keys are converted to a list of {"name": key} or {"name": key, "value": val} dicts sent under the appropriate wire key (vis_params, wget_params, vid_params). The following keys are instead injected directly at the top level of the wire task_params object:

  • return-cookies — send back cookies from the session
  • cookies — Netscape-format cookie file content (string). Top-level for "asset", "single" and "visual" tasks; for "video" it is sent inside vid_params.
  • all_files — include all fetched files alongside the main content. Valid for "visual", "single", and "video" tasks. Raises ValidationError for "asset" tasks. With all_files=True the task result is a directory id, which download_task_result (a plain getfile) cannot fetch; read it through the file API instead. False is dropped, never sent.

Key visual params: output_pdf, output_mhtml, output_image, scale, landscape, paper. See class Attributes for full param sets per task type.

None
run_after int | float | str | None

Optional schedule time — Unix epoch integer (e.g., 1707849000), float epoch, or timezone-aware ISO 8601 string (e.g., "2026-02-13T18:30:00Z" or "2026-02-13T13:30:00-05:00"). Booleans and timezone-less strings are refused. Task enters "wait" status until the specified time.

None
validate_egress bool | None

Look the egress up in its pool's live directory (:meth:check_harvest_egress) and log a warning if it is not listed. Never blocks the task. None (default) uses the LIVE_EGRESS_VALIDATION config key.

None
Note

Wire format parameter mappings:

  • task_type → task_params.request_type
  • urls → task_params.urls
  • task_params (Python dict) → list of {"name": key} or {"name": key, "value": val} dicts sent as:

  • task_params.vis_params for "visual" tasks

  • task_params.wget_params for "asset" tasks
  • task_params.vid_params for "video" tasks

  • egress_info, dest_bucket_id, dest_path, dest_name, dest_auth_token remain at the top-level command dict

  • run_after is injected at top-level command dict alongside egress_info, NOT inside task_params

Wire format structure for visual task::

{
    "command": "create_harvest_task",
    "egress_info": {"name": "New York, NY"},
    "dest_bucket_id": "bucket_abc123",
    "dest_path": "/",
    "dest_name": "capture.pdf",
    "dest_auth_token": "<file_token>",
    "task_params": {
        "request_type": "visual",
        "urls": ["https://example.com"],
        "vis_params": [
            {"name": "output_pdf"},
            {"name": "paper", "value": "A4"}
        ]
    },
    "run_after": 1707849000
}

Returns:

Type Description
str

Task ID string for monitoring and managing the harvest task

Raises:

Type Description
HarvesterAPIError

If task creation fails

ValidationError

If parameters are invalid

Examples:

>>> # Create asset harvest task
>>> task_id = api.create_harvest_task(
...     task_type="asset",
...     urls=["https://example.com/files/"],
...     egress_info={
...         "name": "New York, NY",
...         "categories": {
...             "connectivity": "datacenter",
...             "availability": "private",
...             "protocol": "direct"
...         }
...     },
...     dest_path="/downloads/",
...     dest_name="harvested_files.zip",
...     task_params={
...         "recursive": True,
...         "level": 2,
...         "accept": "*.pdf,*.doc,*.docx"
...     }
... )
>>>
>>> # Create visual PDF capture
>>> task_id = api.create_harvest_task(
...     task_type="visual",  # ← NOT "pdf"
...     urls=["https://example.com/report"],
...     egress_info={"name": "Singapore"},
...     dest_name="report_capture.pdf",
...     task_params={
...         "output_pdf": True,  # ← controls output format
...         "paper": "A4",
...         "landscape": False
...     }
... )
>>>
>>> # Create visual MHTML capture
>>> task_id = api.create_harvest_task(
...     task_type="visual",  # ← NOT "mhtml"
...     urls=["https://example.com/page"],
...     egress_info={"name": "New York, NY"},
...     task_params={"output_mhtml": True}  # ← MHTML format
... )
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.API_COMMAND, api_command="create_harvest_task")
def create_harvest_task(
    self,
    task_type: str,
    urls: list[str],
    egress_info: dict[str, Any],
    dest_path: str = "/",
    dest_name: str | None = None,
    dest_bucket_id: str | None = None,
    task_params: dict[str, Any] | None = None,
    run_after: int | float | str | None = None,
    validate_egress: bool | None = None,
) -> str:
    """Create a new harvest task for web content collection.

    Creates a harvesting task that will collect content from the specified URL
    using the given parameters and store the results in the designated location.

    Args:
        task_type: Type of harvest task. Must be one of ``"asset"``,
            ``"visual"``, ``"video"``, ``"single"``, or ``"wget"``.

            - ``"visual"`` — Full-page capture. Output format controlled by ``task_params``:

              - Screenshot (default): no params, or ``{"output_image": True}``
              - PDF: ``{"output_pdf": True}``
              - MHTML: ``{"output_mhtml": True}``

              **IMPORTANT:** ``"screenshot"``, ``"pdf"``, and ``"mhtml"`` are **not**
              valid ``task_type`` values. Always use ``"visual"`` with ``task_params``.

            - ``"asset"`` — Recursive asset download (HTML, JS, CSS, images).
              Control depth/filters via ``task_params`` (wget-style options).
            - ``"wget"`` — Accepted as its own type and run as an ``"asset"``
              task, with the same parameters.
            - ``"video"`` — Video recording of page interaction (youtube-dl-style options).

        urls: List containing exactly one URL string to harvest.
        egress_info: Egress location dict with required ``"name"`` key
            (e.g., ``{"name": "New York, NY"}``). ``"asset"`` and ``"wget"``
            tasks draw from the six asset datacenter locations (Singapore,
            Dubai, Frankfurt, Sao Paulo, Johannesburg, ``"New York, NY"``):
            pass one of those or ``"World"``. Any other name, such as
            ``"London"``, logs a warning rather than raising, since a
            region or country works when an asset location sits beneath
            it. Optional ``"categories"``
            key accepts ``"connectivity"``, ``"availability"``, ``"protocol"``
            (``direct``, ``tor`` or ``sovereign``); omitted keys are not sent.
            A ``server_set`` key is removed before sending (the server
            rejects it for harvest tasks).
            For non-``"asset"`` task types, ``"connectivity"``/``"protocol"``
            are validated against what is actually available at ``"name"``
            (see ``silo_sdk.harvesting.egress_validation``). Does not
            apply to ``"asset"`` tasks. These checks are advisory. For
            the live answer, see :meth:`list_harvest_egresses` and
            ``validate_egress`` below: ``visual``, ``video`` and
            ``single`` tasks draw from the same pool as browsing
            contexts, ``asset`` and ``wget`` tasks from their own.
        dest_path: Destination path in storage (defaults to ``"/"``)
        dest_name: Name for the result file (auto-generated if not provided)
        dest_bucket_id: Destination bucket ID (uses default if not provided)
        task_params: Task-specific output parameters. Most keys are
            converted to a list of ``{"name": key}`` or
            ``{"name": key, "value": val}`` dicts sent under the
            appropriate wire key (``vis_params``, ``wget_params``,
            ``vid_params``). The following keys are instead injected
            directly at the top level of the wire ``task_params`` object:

            - ``return-cookies`` — send back cookies from the session
            - ``cookies`` — Netscape-format cookie file content (string).
              Top-level for ``"asset"``, ``"single"`` and ``"visual"``
              tasks; for ``"video"`` it is sent inside ``vid_params``.
            - ``all_files`` — include all fetched files alongside the main
              content. Valid for ``"visual"``, ``"single"``, and ``"video"``
              tasks. Raises ``ValidationError`` for ``"asset"`` tasks.
              With ``all_files=True`` the task result is a directory id,
              which ``download_task_result`` (a plain getfile) cannot
              fetch; read it through the file API instead. ``False`` is
              dropped, never sent.


            Key visual params: ``output_pdf``, ``output_mhtml``,
            ``output_image``, ``scale``, ``landscape``, ``paper``.
            See class ``Attributes`` for full param sets per task type.
        run_after: Optional schedule time — Unix epoch integer
            (e.g., ``1707849000``), float epoch, or timezone-aware
            ISO 8601 string (e.g., ``"2026-02-13T18:30:00Z"`` or
            ``"2026-02-13T13:30:00-05:00"``). Booleans and timezone-less
            strings are refused. Task enters ``"wait"``
            status until the specified time.
        validate_egress: Look the egress up in its pool's live directory
            (:meth:`check_harvest_egress`) and log a warning if it is not
            listed. Never blocks the task. ``None`` (default) uses the
            ``LIVE_EGRESS_VALIDATION`` config key.

    Note:
        Wire format parameter mappings:

        - ``task_type`` → ``task_params.request_type``
        - ``urls`` → ``task_params.urls``
        - ``task_params`` (Python dict) → list of ``{"name": key}`` or
          ``{"name": key, "value": val}`` dicts sent as:

          - ``task_params.vis_params`` for ``"visual"`` tasks
          - ``task_params.wget_params`` for ``"asset"`` tasks
          - ``task_params.vid_params`` for ``"video"`` tasks

        - ``egress_info``, ``dest_bucket_id``, ``dest_path``, ``dest_name``,
          ``dest_auth_token`` remain at the top-level command dict
        - ``run_after`` is injected at **top-level command dict** alongside
          ``egress_info``, **NOT** inside ``task_params``

        Wire format structure for visual task::

            {
                "command": "create_harvest_task",
                "egress_info": {"name": "New York, NY"},
                "dest_bucket_id": "bucket_abc123",
                "dest_path": "/",
                "dest_name": "capture.pdf",
                "dest_auth_token": "<file_token>",
                "task_params": {
                    "request_type": "visual",
                    "urls": ["https://example.com"],
                    "vis_params": [
                        {"name": "output_pdf"},
                        {"name": "paper", "value": "A4"}
                    ]
                },
                "run_after": 1707849000
            }

    Returns:
        Task ID string for monitoring and managing the harvest task

    Raises:
        HarvesterAPIError: If task creation fails
        ValidationError: If parameters are invalid

    Examples:
        >>> # Create asset harvest task
        >>> task_id = api.create_harvest_task(
        ...     task_type="asset",
        ...     urls=["https://example.com/files/"],
        ...     egress_info={
        ...         "name": "New York, NY",
        ...         "categories": {
        ...             "connectivity": "datacenter",
        ...             "availability": "private",
        ...             "protocol": "direct"
        ...         }
        ...     },
        ...     dest_path="/downloads/",
        ...     dest_name="harvested_files.zip",
        ...     task_params={
        ...         "recursive": True,
        ...         "level": 2,
        ...         "accept": "*.pdf,*.doc,*.docx"
        ...     }
        ... )
        >>>
        >>> # Create visual PDF capture
        >>> task_id = api.create_harvest_task(
        ...     task_type="visual",  # ← NOT "pdf"
        ...     urls=["https://example.com/report"],
        ...     egress_info={"name": "Singapore"},
        ...     dest_name="report_capture.pdf",
        ...     task_params={
        ...         "output_pdf": True,  # ← controls output format
        ...         "paper": "A4",
        ...         "landscape": False
        ...     }
        ... )
        >>>
        >>> # Create visual MHTML capture
        >>> task_id = api.create_harvest_task(
        ...     task_type="visual",  # ← NOT "mhtml"
        ...     urls=["https://example.com/page"],
        ...     egress_info={"name": "New York, NY"},
        ...     task_params={"output_mhtml": True}  # ← MHTML format
        ... )
    """
    task_params = self._normalize_task_params(task_params)
    top_level_keys = self._top_level_keys_for_type(task_type)

    # Validate all input parameters
    validate_task_creation_params(
        task_type,
        urls,
        egress_info,
        dest_path,
        task_params,
        self.VALID_TASK_TYPES,
        self._valid_params_map,
        top_level_keys=top_level_keys,
    )
    if run_after is not None:
        validate_run_after(run_after)
    if self.live_egress_validation if validate_egress is None else validate_egress:
        self.check_harvest_egress(task_type, egress_info)

    # Prepare normalized data for API
    normalized_egress_info = normalize_egress_info(egress_info)
    bucket_id = dest_bucket_id or self.dest_bucket_id
    final_dest_name = generate_dest_name_if_needed(dest_name, task_type)

    # Build and execute the task creation request
    payload = build_task_creation_payload(
        task_type,
        urls,
        normalized_egress_info,
        bucket_id,
        dest_path,
        final_dest_name,
        task_params or {},
        self.file_api.auth_token,
        run_after=run_after,
        top_level_keys=top_level_keys,
    )

    return self._execute_task_creation(
        task_type, urls, normalized_egress_info, payload
    )

find_harvest_task

find_harvest_task(task_id: str, use_cache: bool = True, finished: bool | None = None) -> dict[str, Any]

Retrieve the status and details of a specific harvest task.

Gets comprehensive information about a harvest task including its current status, timestamps, results, and any error information. Supports caching to reduce API calls for frequently polled tasks.

Parameters:

Name Type Description Default
task_id str

The ID of the task to retrieve

required
use_cache bool

Whether to use cached results (default: True)

True
finished bool | None

If True, return only completed tasks. If False, return only in-progress tasks. If omitted, return regardless of completion state.

None

Returns:

Name Type Description
dict[str, Any]

The task record as the platform returns it. This method passes

dict[str, Any]

the record straight through — it does not rename or normalize

dict[str, Any]

anything — so the timestamps are create_ts/start_ts/

dict[str, Any]

end_ts and the result payload is task_results. The

dict[str, Any]

created_ts/started_ts/completed_ts/task_metadata

dict[str, Any]

aliases exist only on the dictionaries returned by

dict[str, Any]

wait_for_completion() and wait_for_completion_async(),

dict[str, Any]

which layer them on top of these fields.

dict[str, Any]

Identity and state:

dict[str, Any]
  • task_id: Task identifier
dict[str, Any]
  • status: One of wait, claimed, done or error
dict[str, Any]
  • finished: True once the task has been resolved, successfully or not. A finished task is never claimed or retried again
dict[str, Any]
  • attempt_count: Attempts made so far, counting the attempt currently in flight. A task stays eligible to run while this is below the platform's attempt limit
dict[str, Any]
  • last_error: Failure message, populated when status is error. Also set when a task is failed for exhausting its retry limit
dict[str, Any]
  • hostname: The worker that most recently claimed the task. This identifies the machine doing the harvesting, not the target
dict[str, Any]
  • org_id: Organization that owns the task. Assigned by the platform at creation and not accepted from the caller
dict[str, Any]

Timing, as Unix epoch floats or null:

dict[str, Any]
  • create_ts: When the task was created
dict[str, Any]
  • start_ts: When a worker most recently claimed it. Null until the first claim, and reset on each subsequent attempt
dict[str, Any]
  • end_ts: When the task was resolved. Null until then, and cleared again if the task is re-attempted
dict[str, Any]
  • run_after: Earliest time the task may be claimed. Set from the run_after creation argument and pushed forward whenever a worker reschedules a retry
dict[str, Any]

What was requested:

dict[str, Any]
  • request_type: Task type. The platform strips and lower-cases this, so it can come back in a different case than it was sent
dict[str, Any]
  • target: Worker pool the task is routed to. Only workers polling for the same target can claim it
dict[str, Any]
  • name: Caller-supplied label, echoed back unchanged
dict[str, Any]
  • task_params: The per-type parameters the task was created with, including urls. The platform reflects request_type into this dictionary and removes the internal log token from it before returning
dict[str, Any]
  • dest_bucket_id: Storage bucket the results are written to
dict[str, Any]
  • dest_path: Path within that bucket
dict[str, Any]
  • dest_name: Name of the result object
dict[str, Any]
  • egress_info: Egress selection in effect, as {"name": ..., "categories": {...}}
dict[str, Any]
  • egress_tag: Superseded egress selector, kept for older integrations. Null whenever egress_info is set, which is every task this SDK creates
Results dict[str, Any]
dict[str, Any]
  • result_file_id: File ID of the result, populated once the task completes successfully
dict[str, Any]
  • task_results: Raw result payload from the worker. Task metadata is nested inside it under the task-metadata key
Note dict[str, Any]

There is no progress field. The platform reports a task's state through status and finished only; it does not report a percentage. An earlier version of this docstring listed progress "if available", and it is never available — anything reading it gets None every time.

running is rarely, if ever, observed. The documented statuses are wait, claimed, done and error; a task in flight normally reads claimed. The worker code does write running, so keep code that branches on status tolerant of it.

Raises:

Type Description
HarvesterAPIError

If task retrieval fails

ValidationError

If task_id is invalid

Examples:

>>> task_info = api.find_harvest_task("task_123")
>>> print(f"Status: {task_info['status']}")
>>> if task_info['status'] == 'done':
...     print(f"Result file: {task_info['result_file_id']}")
>>> elif task_info['status'] == 'error':
...     print(f"Error: {task_info['last_error']}")
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.API_COMMAND, api_command="find_harvest_task")
def find_harvest_task(
    self,
    task_id: str,
    use_cache: bool = True,
    finished: bool | None = None,
) -> dict[str, Any]:
    """Retrieve the status and details of a specific harvest task.

    Gets comprehensive information about a harvest task including its current
    status, timestamps, results, and any error information. Supports caching
    to reduce API calls for frequently polled tasks.

    Args:
        task_id: The ID of the task to retrieve
        use_cache: Whether to use cached results (default: True)
        finished: If True, return only completed tasks. If False, return only
            in-progress tasks. If omitted, return regardless of completion state.

    Returns:
        The task record as the platform returns it. This method passes
        the record straight through — it does not rename or normalize
        anything — so the timestamps are ``create_ts``/``start_ts``/
        ``end_ts`` and the result payload is ``task_results``. The
        ``created_ts``/``started_ts``/``completed_ts``/``task_metadata``
        aliases exist only on the dictionaries returned by
        ``wait_for_completion()`` and ``wait_for_completion_async()``,
        which layer them on top of these fields.

        Identity and state:

        - task_id: Task identifier
        - status: One of ``wait``, ``claimed``, ``done`` or ``error``
        - finished: True once the task has been resolved, successfully or
            not. A finished task is never claimed or retried again
        - attempt_count: Attempts made so far, **counting the attempt
            currently in flight**. A task stays eligible to run while this
            is below the platform's attempt limit
        - last_error: Failure message, populated when ``status`` is
            ``error``. Also set when a task is failed for exhausting its
            retry limit
        - hostname: The worker that most recently claimed the task. This
            identifies the machine doing the harvesting, not the target
        - org_id: Organization that owns the task. Assigned by the
            platform at creation and not accepted from the caller

        Timing, as Unix epoch floats or null:

        - create_ts: When the task was created
        - start_ts: When a worker most recently claimed it. Null until
            the first claim, and reset on each subsequent attempt
        - end_ts: When the task was resolved. Null until then, and
            cleared again if the task is re-attempted
        - run_after: Earliest time the task may be claimed. Set from the
            ``run_after`` creation argument and pushed forward whenever a
            worker reschedules a retry

        What was requested:

        - request_type: Task type. The platform strips and lower-cases
            this, so it can come back in a different case than it was sent
        - target: Worker pool the task is routed to. Only workers polling
            for the same target can claim it
        - name: Caller-supplied label, echoed back unchanged
        - task_params: The per-type parameters the task was created with,
            including ``urls``. The platform reflects ``request_type`` into
            this dictionary and removes the internal log token from it
            before returning
        - dest_bucket_id: Storage bucket the results are written to
        - dest_path: Path within that bucket
        - dest_name: Name of the result object
        - egress_info: Egress selection in effect, as ``{"name": ...,
            "categories": {...}}``
        - egress_tag: Superseded egress selector, kept for older
            integrations. Null whenever ``egress_info`` is set, which is
            every task this SDK creates

        Results:

        - result_file_id: File ID of the result, populated once the task
            completes successfully
        - task_results: Raw result payload from the worker. Task metadata
            is nested inside it under the ``task-metadata`` key

        Note:
            **There is no progress field.** The platform reports a task's
            state through ``status`` and ``finished`` only; it does not
            report a percentage. An earlier version of this docstring
            listed ``progress`` "if available", and it is never available —
            anything reading it gets ``None`` every time.

            **``running`` is rarely, if ever, observed.** The documented
            statuses are ``wait``, ``claimed``, ``done`` and ``error``; a
            task in flight normally reads ``claimed``. The worker code
            does write ``running``, so keep code that branches on status
            tolerant of it.

    Raises:
        HarvesterAPIError: If task retrieval fails
        ValidationError: If task_id is invalid

    Examples:
        >>> task_info = api.find_harvest_task("task_123")
        >>> print(f"Status: {task_info['status']}")
        >>> if task_info['status'] == 'done':
        ...     print(f"Result file: {task_info['result_file_id']}")
        >>> elif task_info['status'] == 'error':
        ...     print(f"Error: {task_info['last_error']}")
    """
    self._require_task_id(task_id)

    # Check cache if enabled and requested
    # The cache is keyed by task id only, so a record cached without the
    # filter must not answer a filtered query.
    if (
        use_cache
        and finished is None
        and self._status_cache_enabled
        and self._status_cache is not None
    ):
        cached_result = get_cached_task_status(
            self._status_cache, task_id, self._status_cache_ttl
        )
        if cached_result is not None:
            self.logger.debug("Retrieved task status from cache: %s", task_id)
            return cached_result

    command: dict[str, Any] = {"command": "find_harvest_task", "task_id": task_id}
    if finished is not None:
        command["finished"] = finished
    payload = [command]

    try:
        self.logger.debug("Finding harvest task: %s", task_id)
        response = self._make_api_request("POST", "api/", self.auth_token, payload)

        # Debug logging for response structure (only in debug mode)
        self.logger.debug("Processing response for task: %s", task_id)

        # Handle different response formats more robustly
        try:
            result = self._extract_api_result(response)
        except SiloError as extract_error:
            # A reported API error is a real answer, not a parse failure.
            #
            # _extract_api_result raises SiloError for both shapes the
            # endpoint uses to signal failure: {"error": ...} in place of
            # result, and {"result": {"error": ...}} nested inside one,
            # and attaches the offending element as response_data. The Eng
            # capture has a recorded example of the first on this very
            # command ("Error: invalid auth token"). Falling back to
            # response[1] here handed that envelope back as if it were a
            # task: callers got {"error": ...} whose .get("status") is
            # None, it was written into the status cache, and
            # wait_for_completion reported "Unexpected task status: None"
            # instead of the server's message. Let it through.
            if extract_error.response_data is not None:
                raise
            # No response_data means the shape was malformed rather than
            # an error report: fall back to direct response processing.
            self.logger.debug(
                "Standard extraction failed, trying direct response processing: %s",
                extract_error,
            )
            if isinstance(response, list) and len(response) > 1:
                result = response[1]
            else:
                raise HarvesterAPIError(
                    f"Unexpected response format: {response}"
                ) from extract_error

        # Handle different result formats
        if isinstance(result, list) and len(result) > 0:
            task_details = result[0]
        elif isinstance(result, dict):
            task_details = result
        elif isinstance(result, str):
            # Handle string responses that might contain error information
            if "task produced no output" in result or "TypeError" in result:
                # Preserve the original server error message without additional wrapping
                raise HarvesterAPIError(
                    f"Task {task_id} failed with server error: {result}"
                )
            else:
                raise HarvesterAPIError(
                    f"Unexpected string response for task_id: {task_id} - {result}"
                )
        else:
            raise HarvesterAPIError(
                f"No task details found for task_id: {task_id} - "
                f"got result type: {type(result)}"
            )

        # Ensure task_details is a dictionary
        if not isinstance(task_details, dict):
            # Handle string task details that might contain error information
            if isinstance(task_details, str) and (
                "task produced no output" in task_details
                or "TypeError" in task_details
            ):
                raise HarvesterAPIError(
                    f"Task {task_id} failed with server error: {task_details}"
                )
            else:
                raise HarvesterAPIError(
                    f"Task details must be a dictionary, "
                    f"got {type(task_details)}: {task_details}"
                )

        self.logger.debug("Retrieved task details for: %s", task_id)

        # Cache the result if caching is enabled
        if self._status_cache_enabled:
            cache_task_status(
                self._status_cache, task_id, task_details, self._status_cache_ttl
            )

        return task_details

    except Exception as e:
        if isinstance(e, HarvesterAPIError):
            raise
        raise HarvesterAPIError(f"Failed to find harvest task: {e}") from e

delete_harvest_task

delete_harvest_task(task_id: str) -> bool

Delete a specific harvest task.

Permanently removes a harvest task and its associated data. This action cannot be undone. If the task is still running, the worker aborts it at its next lock extension, not instantly.

Parameters:

Name Type Description Default
task_id str

The ID of the task to delete

required

Returns:

Type Description
bool

True if deletion was successful, False otherwise

Raises:

Type Description
HarvesterAPIError

If task deletion fails

ValidationError

If task_id is invalid

Examples:

>>> success = api.delete_harvest_task("task_123")
>>> if success:
...     print("Task deleted successfully")
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.API_COMMAND, api_command="delete_harvest_task")
def delete_harvest_task(self, task_id: str) -> bool:
    """Delete a specific harvest task.

    Permanently removes a harvest task and its associated data. This action
    cannot be undone. If the task is still running, the worker aborts it at
    its next lock extension, not instantly.

    Args:
        task_id: The ID of the task to delete

    Returns:
        True if deletion was successful, False otherwise

    Raises:
        HarvesterAPIError: If task deletion fails
        ValidationError: If task_id is invalid

    Examples:
        >>> success = api.delete_harvest_task("task_123")
        >>> if success:
        ...     print("Task deleted successfully")
    """
    self._require_task_id(task_id)

    payload = [{"command": "delete_harvest_task", "task_id": task_id}]

    try:
        self.logger.debug("Deleting harvest task: %s", task_id)
        response = self._make_api_request("POST", "api/", self.auth_token, payload)

        result = self._extract_api_result(response)
        if isinstance(result, dict) and result.get("deleted"):
            self.logger.info("Successfully deleted harvest task: %s", task_id)
            return True

        self.logger.warning("Task deletion may have failed: %s", result)
        return False

    except Exception as e:
        if isinstance(e, HarvesterAPIError):
            raise
        raise HarvesterAPIError(f"Failed to delete harvest task: {e}") from e

list_harvest_egresses

list_harvest_egresses(task_type: str = 'visual', *, org_id: str | int | None = None, org_name: str | None = None) -> list[EgressDirectoryEntry]

List the egress a harvest task of this type can draw from.

Harvest tasks resolve egress for the organization the harvest token administers, from a pool chosen by task type: asset and wget use the asset pool, visual, video and single the pool browsing contexts use. This asks the platform's directory for that pool and org.

The org defaults to HARVEST_EGRESS_ORG, or TOP_ORG when that is unset. Harvest resolves egress for the org the harvest token belongs to, and an org lookup sees only egress provisioned to that org and the orgs above it. When the harvest token belongs to a sub-org of TOP_ORG (common: egress provisioned to a workspace sub-org), set HARVEST_EGRESS_ORG to that sub-org; otherwise the lookup sees nothing and the live check is skipped.

The answer can under-report: an org's Launcher Options hide regions from the directory but not from harvest, so a task may still succeed with an egress this list omits. An empty list does not mean harvest will fail: an org whose egress is provisioned to its sub-orgs lists nothing, so query the sub-org instead.

Parameters:

Name Type Description Default
task_type str

The harvest task type the egress is for.

'visual'
org_id str | int | None

The org to report for. Defaults to HARVEST_EGRESS_ORG, or TOP_ORG if that is unset.

None
org_name str | None

The org by name, as an alternative to org_id.

None

Returns:

Type Description
list[EgressDirectoryEntry]

Directory entries, as :meth:BrowsingAPI.list_egresses returns.

Raises:

Type Description
ValidationError

If task_type is not a harvest task type.

ConfigurationError

Without ADMIN_TOKEN (the directory needs an admin token), or with no org and neither HARVEST_EGRESS_ORG nor TOP_ORG.

BrowsingAPIError

If the lookup fails.

Examples:

>>> pool = harvester.list_harvest_egresses("visual")
>>> [e["name"] for e in pool]
['World', 'Europe', 'Germany', 'Frankfurt', ...]
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.CONVENIENCE, wraps=["list_egresses"])
def list_harvest_egresses(
    self,
    task_type: str = "visual",
    *,
    org_id: str | int | None = None,
    org_name: str | None = None,
) -> list[EgressDirectoryEntry]:
    """List the egress a harvest task of this type can draw from.

    Harvest tasks resolve egress for the organization the harvest token
    administers, from a pool chosen by task type: ``asset`` and ``wget``
    use the asset pool, ``visual``, ``video`` and ``single`` the pool
    browsing contexts use. This asks the platform's directory for that
    pool and org.

    The org defaults to ``HARVEST_EGRESS_ORG``, or ``TOP_ORG`` when that
    is unset. Harvest resolves egress for the org the harvest token
    belongs to, and an org lookup sees only egress provisioned to that
    org and the orgs above it. When the harvest token belongs to a
    sub-org of ``TOP_ORG`` (common: egress provisioned to a workspace
    sub-org), set ``HARVEST_EGRESS_ORG`` to that sub-org; otherwise the
    lookup sees nothing and the live check is skipped.

    The answer can **under-report**: an org's Launcher Options hide
    regions from the directory but not from harvest, so a task may still
    succeed with an egress this list omits. An empty list does not mean
    harvest will fail: an org whose egress is provisioned to its
    sub-orgs lists nothing, so query the sub-org instead.

    Args:
        task_type: The harvest task type the egress is for.
        org_id: The org to report for. Defaults to
            ``HARVEST_EGRESS_ORG``, or ``TOP_ORG`` if that is unset.
        org_name: The org by name, as an alternative to ``org_id``.

    Returns:
        Directory entries, as :meth:`BrowsingAPI.list_egresses` returns.

    Raises:
        ValidationError: If ``task_type`` is not a harvest task type.
        ConfigurationError: Without ``ADMIN_TOKEN`` (the directory needs
            an admin token), or with no org and neither
            ``HARVEST_EGRESS_ORG`` nor ``TOP_ORG``.
        BrowsingAPIError: If the lookup fails.

    Examples:
        >>> pool = harvester.list_harvest_egresses("visual")
        >>> [e["name"] for e in pool]
        ['World', 'Europe', 'Germany', 'Frankfurt', ...]
    """
    if task_type not in self.VALID_TASK_TYPES:
        raise ValidationError(
            f"Invalid task_type: {task_type}. "
            f"Valid types are: {', '.join(sorted(self.VALID_TASK_TYPES))}"
        )
    if org_id is None and not org_name:
        org_name = self._harvest_egress_org() or None
        if not org_name:
            raise ConfigurationError(
                "list_harvest_egresses needs org_id or org_name, or "
                "HARVEST_EGRESS_ORG or TOP_ORG in the configuration (set "
                "A8_HARVEST_EGRESS_ORG or A8_TOP_ORG)"
            )
    return cast(
        "list[EgressDirectoryEntry]",
        self._browsing_api().list_egresses(
            org_id=org_id,
            org_name=org_name,
            session_type=HARVEST_DIRECTORY_SESSION_TYPE,
            service_type=harvest_service_type(task_type),
        ),
    )

check_harvest_egress

check_harvest_egress(task_type: str, egress_info: dict[str, Any], *, warn: bool = True) -> str | None

Check a harvest egress against its pool's directory; never raises.

Looks the egress up in :meth:list_harvest_egresses for HARVEST_EGRESS_ORG, or TOP_ORG if that is unset, and logs a warning when it is not listed. A miss does not block the task, because the directory can under-report (see :meth:list_harvest_egresses). Lookups are reused for a minute per pool, so a batch asks once. A failed lookup is remembered for that minute too, so a batch logs it once instead of retrying per task. Safe to call from several threads: concurrent checks of one pool share a single lookup, and checks of different pools never wait on each other.

Parameters:

Name Type Description Default
task_type str

The harvest task type.

required
egress_info dict[str, Any]

The task's egress selection.

required
warn bool

Log the gap as a warning saying the task was not blocked. Pass False when the caller acts on the returned gap itself, for example by refusing the task. Other logs (a skipped lookup, an empty pool) are unaffected.

True

Returns:

Type Description
str | None

A description of the gap, or None when the egress is listed or

str | None

the check could not run (no ADMIN_TOKEN, neither

str | None

HARVEST_EGRESS_ORG nor TOP_ORG, or a failed lookup — each

str | None

logged).

Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.CONVENIENCE, wraps=["list_egresses"])
def check_harvest_egress(
    self, task_type: str, egress_info: dict[str, Any], *, warn: bool = True
) -> str | None:
    """Check a harvest egress against its pool's directory; never raises.

    Looks the egress up in :meth:`list_harvest_egresses` for
    ``HARVEST_EGRESS_ORG``, or ``TOP_ORG`` if that is unset, and logs a
    warning when it is not listed. A miss does not block the task,
    because the directory can under-report (see
    :meth:`list_harvest_egresses`). Lookups are reused for a minute per
    pool, so a batch asks once. A failed lookup is remembered for that
    minute too, so a batch logs it once instead of retrying per task.
    Safe to call from several threads: concurrent checks of one pool
    share a single lookup, and checks of different pools never wait on
    each other.

    Args:
        task_type: The harvest task type.
        egress_info: The task's egress selection.
        warn: Log the gap as a warning saying the task was not blocked.
            Pass ``False`` when the caller acts on the returned gap
            itself, for example by refusing the task. Other logs (a
            skipped lookup, an empty pool) are unaffected.

    Returns:
        A description of the gap, or None when the egress is listed or
        the check could not run (no ``ADMIN_TOKEN``, neither
        ``HARVEST_EGRESS_ORG`` nor ``TOP_ORG``, or a failed lookup — each
        logged).
    """
    if (
        task_type not in self.VALID_TASK_TYPES
        or not isinstance(egress_info, dict)
        or not egress_info.get("name")
    ):
        # Shape and type errors are create_harvest_task's to report.
        return None
    org_name = self._harvest_egress_org()
    key = (harvest_service_type(task_type), org_name)
    directory: list[EgressDirectoryEntry] | None
    with self._pool_lock(key):
        now = time.monotonic()
        cached = self._directory_cache.get(key)
        if cached is not None and now - cached[0] < _DIRECTORY_TTL_SECONDS:
            directory = cached[1]
        else:
            try:
                directory = self.list_harvest_egresses(task_type)
            except (BrowsingAPIError, ConfigurationError) as e:
                self.logger.warning(
                    "Live harvest egress check skipped: %s. The task has NOT "
                    "been blocked, and the check will not be retried for "
                    "%d seconds.",
                    e,
                    int(_DIRECTORY_TTL_SECONDS),
                )
                directory = None
            self._directory_cache[key] = (now, directory)
    if directory is None:
        return None
    if not directory:
        self.logger.info(
            "Live harvest egress check: the directory lists nothing in the "
            "%s pool for org %r, so there is nothing to check against. An "
            "org lookup sees only egress provisioned to that org and the "
            "orgs above it, so an org whose egress is provisioned to its "
            "sub-orgs lists nothing: set HARVEST_EGRESS_ORG to the sub-org "
            "the harvest token belongs to. Not checking.",
            key[0],
            org_name,
        )
        return None
    gap = describe_harvest_egress_gap(egress_info, directory)
    if gap and warn:
        self.logger.warning(
            "Live harvest egress check: %s. The task has NOT been blocked: "
            "harvest can use egress an org's Launcher Options hide from "
            "this list.",
            gap.rstrip("."),
        )
    return gap

wait_for_completion

wait_for_completion(task_id: str, max_wait_time: int = 1800, check_interval: int = 10, progress_callback: Callable[..., Any] | None = None) -> dict[str, Any]

Wait for a harvest task to complete, with optional progress monitoring.

Monitors a harvest task until it completes successfully, encounters an error, or the maximum wait time is exceeded. Provides optional progress callbacks for real-time monitoring.

Parameters:

Name Type Description Default
task_id str

The ID of the task to monitor

required
max_wait_time int

Maximum time to wait in seconds (default: 30 minutes). A deadline: the last status check happens at it, not one check_interval past it.

1800
check_interval int

Time between status checks in seconds (default: 10)

10
progress_callback Callable[..., Any] | None

Optional function to call with progress updates

None

Returns:

Type Description
dict[str, Any]

The final task record, exactly as find_harvest_task()

dict[str, Any]

returned it on the last poll — this method hands that

dict[str, Any]

dictionary straight back and neither renames nor adds any

dict[str, Any]

field. See find_harvest_task() for the full field list;

dict[str, Any]

in particular the timestamps are create_ts/start_ts/

dict[str, Any]

end_ts, and the created_ts/started_ts/

dict[str, Any]

completed_ts aliases are not present here. Only

dict[str, Any]

wait_for_completion_async() adds those.

Raises:

Type Description
HarvesterAPIError

If the task fails or times out. A task that reaches error status is raised, not returned, with the record's last_error in the message.

ValidationError

If parameters are invalid

Examples:

>>> def progress_handler(task_info):
...     # No percentage is available; status is the signal.
...     print(f"Status: {task_info['status']}")
>>>
>>> result = api.wait_for_completion(
...     task_id="task_123",
...     max_wait_time=600,  # 10 minutes
...     progress_callback=progress_handler
... )
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.CONVENIENCE, wraps=["find_harvest_task"])
def wait_for_completion(
    self,
    task_id: str,
    max_wait_time: int = 1800,  # 30 minutes
    check_interval: int = 10,
    progress_callback: Callable[..., Any] | None = None,
) -> dict[str, Any]:
    """Wait for a harvest task to complete, with optional progress monitoring.

    Monitors a harvest task until it completes successfully, encounters an error,
    or the maximum wait time is exceeded. Provides optional progress callbacks
    for real-time monitoring.

    Args:
        task_id: The ID of the task to monitor
        max_wait_time: Maximum time to wait in seconds (default: 30 minutes).
            A deadline: the last status check happens at it, not one
            ``check_interval`` past it.
        check_interval: Time between status checks in seconds (default: 10)
        progress_callback: Optional function to call with progress updates

    Returns:
        The final task record, exactly as ``find_harvest_task()``
        returned it on the last poll — this method hands that
        dictionary straight back and neither renames nor adds any
        field. See ``find_harvest_task()`` for the full field list;
        in particular the timestamps are ``create_ts``/``start_ts``/
        ``end_ts``, and the ``created_ts``/``started_ts``/
        ``completed_ts`` aliases are **not** present here. Only
        ``wait_for_completion_async()`` adds those.

    Raises:
        HarvesterAPIError: If the task fails or times out. A task
            that reaches ``error`` status is raised, not returned,
            with the record's ``last_error`` in the message.
        ValidationError: If parameters are invalid

    Examples:
        >>> def progress_handler(task_info):
        ...     # No percentage is available; status is the signal.
        ...     print(f"Status: {task_info['status']}")
        >>>
        >>> result = api.wait_for_completion(
        ...     task_id="task_123",
        ...     max_wait_time=600,  # 10 minutes
        ...     progress_callback=progress_handler
        ... )
    """
    self._require_task_id(task_id)

    if not isinstance(max_wait_time, int) or max_wait_time <= 0:
        raise ValidationError("max_wait_time must be a positive integer")

    if not isinstance(check_interval, int) or check_interval <= 0:
        raise ValidationError("check_interval must be a positive integer")

    start_time = time.time()
    checks_performed = 0

    self.logger.info(
        "Waiting for task completion: %s (max %d seconds)", task_id, max_wait_time
    )

    try:
        while True:
            # Check if we've exceeded the maximum wait time
            elapsed_time = time.time() - start_time
            if elapsed_time > max_wait_time:
                raise HarvesterAPIError(
                    f"Task {task_id} did not complete within {max_wait_time} seconds"
                )

            # Get current task status
            try:
                task_status = self.find_harvest_task(task_id)
                if not isinstance(task_status, dict):
                    raise HarvesterAPIError(
                        f"Invalid task status format: {type(task_status)}"
                    )

                status = task_status.get("status")
                checks_performed += 1

                # Call progress callback if provided
                if progress_callback is not None:
                    try:
                        progress_callback(task_status)
                    except Exception as e:
                        self.logger.warning("Progress callback failed: %s", e)

                # Check terminal states
                if status == "done":
                    self.logger.info(
                        "Task %s completed successfully after %d checks (%.1f seconds)",
                        task_id,
                        checks_performed,
                        elapsed_time,
                    )
                    return task_status

                elif status == "error":
                    error_msg = task_status.get("last_error", "Unknown error")
                    raise HarvesterAPIError(
                        f"Task {task_id} failed with server error: {error_msg}"
                    )

            except HarvesterAPIError as e:
                # Re-check the original error message format
                error_str = str(e)
                if (
                    "task produced no output" in error_str
                    and "TypeError" in error_str
                ):
                    # This indicates a response parsing issue from find_harvest_task
                    self.logger.error(
                        "Task status retrieval failed due to response format issue: %s",
                        e,
                    )
                    # Don't re-wrap the error message - just re-raise the original
                    raise
                else:
                    raise

            if status in IN_PROGRESS_TASK_STATUSES:
                # Task is still in progress
                self.logger.debug(
                    "Task %s status: %s (check %d, %.1f seconds elapsed)",
                    task_id,
                    status,
                    checks_performed,
                    elapsed_time,
                )
                delay = _poll_delay(check_interval, max_wait_time, elapsed_time)
                if delay is None:
                    raise HarvesterAPIError(
                        f"Task {task_id} did not complete within "
                        f"{max_wait_time} seconds"
                    )
                time.sleep(delay)

            elif status not in ["done", "error"]:
                raise HarvesterAPIError(f"Unexpected task status: {status}")

    except KeyboardInterrupt as exc:
        self.logger.info("Task monitoring interrupted by user")
        raise HarvesterAPIError("Task monitoring was interrupted") from exc

wait_for_completion_with_progress

wait_for_completion_with_progress(task_id: str, task_type: str, max_wait_time: int = 1800, check_interval: int = 10, progress_callback: Callable[..., Any] | None = None) -> TaskProgress

Wait for a harvest task to complete with enhanced progress tracking.

Monitors a harvest task with detailed progress metrics including timing, status transitions, and estimated completion times.

Parameters:

Name Type Description Default
task_id str

The ID of the task to monitor

required
task_type str

Type of harvest task

required
max_wait_time int

Maximum time to wait in seconds (default: 30 minutes). A deadline: the last status check happens at it, not one check_interval past it.

1800
check_interval int

Time between status checks in seconds (default: 10)

10
progress_callback Callable[..., Any] | None

Optional function to call with TaskProgress updates

None

Returns:

Type Description
TaskProgress

TaskProgress object with detailed metrics when completed

Raises:

Type Description
HarvesterAPIError

If task fails or times out

ValidationError

If parameters are invalid

Examples:

>>> def progress_handler(progress: TaskProgress):
...     print(f"Status: {progress.status} ({progress.progress_percentage}%)")
...     if progress.estimated_completion:
...         eta = datetime.fromtimestamp(progress.estimated_completion)
...         print(f"ETA: {eta}")
>>>
>>> progress = api.wait_for_completion_with_progress(
...     task_id="task_123",
...     task_type="asset",
...     progress_callback=progress_handler
... )
>>> print(f"Completed in {progress.get_duration():.1f} seconds")
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.CONVENIENCE, wraps=["find_harvest_task"])
def wait_for_completion_with_progress(
    self,
    task_id: str,
    task_type: str,
    max_wait_time: int = 1800,  # 30 minutes
    check_interval: int = 10,
    progress_callback: Callable[..., Any] | None = None,
) -> TaskProgress:
    """Wait for a harvest task to complete with enhanced progress tracking.

    Monitors a harvest task with detailed progress metrics including timing,
    status transitions, and estimated completion times.

    Args:
        task_id: The ID of the task to monitor
        task_type: Type of harvest task
        max_wait_time: Maximum time to wait in seconds (default: 30 minutes).
            A deadline: the last status check happens at it, not one
            ``check_interval`` past it.
        check_interval: Time between status checks in seconds (default: 10)
        progress_callback: Optional function to call with TaskProgress updates

    Returns:
        TaskProgress object with detailed metrics when completed

    Raises:
        HarvesterAPIError: If task fails or times out
        ValidationError: If parameters are invalid

    Examples:
        >>> def progress_handler(progress: TaskProgress):
        ...     print(f"Status: {progress.status} ({progress.progress_percentage}%)")
        ...     if progress.estimated_completion:
        ...         eta = datetime.fromtimestamp(progress.estimated_completion)
        ...         print(f"ETA: {eta}")
        >>>
        >>> progress = api.wait_for_completion_with_progress(
        ...     task_id="task_123",
        ...     task_type="asset",
        ...     progress_callback=progress_handler
        ... )
        >>> print(f"Completed in {progress.get_duration():.1f} seconds")
    """
    self._require_task_id(task_id)

    if not isinstance(max_wait_time, int) or max_wait_time <= 0:
        raise ValidationError("max_wait_time must be a positive integer")

    if not isinstance(check_interval, int) or check_interval <= 0:
        raise ValidationError("check_interval must be a positive integer")

    # Initialize progress tracker
    progress = TaskProgress(task_id, task_type)

    start_time = time.time()
    checks_performed = 0

    self.logger.info(
        "Waiting for task completion with progress tracking: %s (max %d seconds)",
        task_id,
        max_wait_time,
    )

    try:
        while True:
            # Check if we've exceeded the maximum wait time
            elapsed_time = time.time() - start_time
            if elapsed_time > max_wait_time:
                progress.set_error(f"Task timeout after {max_wait_time} seconds")
                raise HarvesterAPIError(
                    f"Task {task_id} did not complete within {max_wait_time} seconds"
                )

            # Get current task status
            task_status = self.find_harvest_task(task_id)
            status: str = task_status.get("status", "unknown")
            checks_performed += 1

            # Update progress tracker
            # No progress argument: the platform reports state through
            # status and finished and sends no percentage, so this was
            # always None -- and it was being passed positionally into
            # update_status' "message" parameter, not its "progress" one.
            # Two mistakes that cancelled out only because the value never
            # existed.
            progress.update_status(status)
            progress.retry_count = checks_performed - 1

            # Call progress callback if provided
            if progress_callback is not None:
                try:
                    progress_callback(progress)
                except Exception as e:
                    self.logger.warning("Progress callback failed: %s", e)

            # Check terminal states
            if status == "done":
                self.logger.info(
                    "Task %s completed successfully after %d checks (%.1f seconds)",
                    task_id,
                    checks_performed,
                    elapsed_time,
                )
                return progress

            elif status == "error":
                error_msg = task_status.get("last_error", "Unknown error")
                progress.set_error(error_msg)
                raise HarvesterAPIError(
                    f"Task {task_id} failed with server error: {error_msg}"
                )

            elif status in IN_PROGRESS_TASK_STATUSES:
                # Task is still in progress
                self.logger.debug(
                    "Task %s status: %s (check %d, %.1f seconds elapsed)",
                    task_id,
                    status,
                    checks_performed,
                    elapsed_time,
                )
                delay = _poll_delay(check_interval, max_wait_time, elapsed_time)
                if delay is None:
                    progress.set_error(
                        f"Task timeout after {max_wait_time} seconds"
                    )
                    raise HarvesterAPIError(
                        f"Task {task_id} did not complete within "
                        f"{max_wait_time} seconds"
                    )
                time.sleep(delay)

            else:
                progress.set_error(f"Unexpected task status: {status}")
                raise HarvesterAPIError(f"Unexpected task status: {status}")

    except KeyboardInterrupt as exc:
        progress.set_error("Task monitoring interrupted by user")
        self.logger.info("Task monitoring interrupted by user")
        raise HarvesterAPIError("Task monitoring was interrupted") from exc

download_task_result

download_task_result(task_id: str, output_path: str, verify_completion: bool = True) -> str

Download the result of a completed harvest task.

Downloads the harvested content from a completed task to the local filesystem. Optionally verifies that the task is completed before attempting download.

Parameters:

Name Type Description Default
task_id str

The ID of the completed task

required
output_path str

Local path where to save the downloaded file

required
verify_completion bool

Whether to verify task completion before download

True

Returns:

Type Description
str

Path of the downloaded file

Raises:

Type Description
HarvesterAPIError

If download fails or task is not completed

ValidationError

If parameters are invalid

Examples:

>>> downloaded_file = api.download_task_result(
...     task_id="task_123",
...     output_path="/local/downloads/harvest_result.zip"
... )
>>> print(f"Downloaded to: {downloaded_file}")
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.CONVENIENCE, wraps=["find_harvest_task", "getfile"])
def download_task_result(
    self,
    task_id: str,
    output_path: str,
    verify_completion: bool = True,
) -> str:
    """Download the result of a completed harvest task.

    Downloads the harvested content from a completed task to the local filesystem.
    Optionally verifies that the task is completed before attempting download.

    Args:
        task_id: The ID of the completed task
        output_path: Local path where to save the downloaded file
        verify_completion: Whether to verify task completion before download

    Returns:
        Path of the downloaded file

    Raises:
        HarvesterAPIError: If download fails or task is not completed
        ValidationError: If parameters are invalid

    Examples:
        >>> downloaded_file = api.download_task_result(
        ...     task_id="task_123",
        ...     output_path="/local/downloads/harvest_result.zip"
        ... )
        >>> print(f"Downloaded to: {downloaded_file}")
    """
    self._require_task_id(task_id)

    if not output_path:
        raise ValidationError("output_path must be provided")

    # Verify task completion if requested
    if verify_completion:
        task_status = self.find_harvest_task(task_id)
        if task_status.get("status") != "done":
            raise HarvesterAPIError(
                f"Task {task_id} is not completed. "
                f"Current status: {task_status.get('status')}"
            )

        result_file_id = task_status.get("result_file_id")
        if not result_file_id:
            raise HarvesterAPIError(f"No result file found for task {task_id}")
    else:
        # Get task details to find result file ID
        task_status = self.find_harvest_task(task_id)
        result_file_id = task_status.get("result_file_id")
        if not result_file_id:
            raise HarvesterAPIError(f"No result file found for task {task_id}")

    try:
        self.logger.debug("Downloading result for task: %s", task_id)
        downloaded_path = self.file_api.download_file(result_file_id, output_path)

        self.logger.info(
            "Successfully downloaded task result: %s -> %s",
            task_id,
            downloaded_path,
        )
        return downloaded_path

    except FileAPIError as e:
        raise HarvesterAPIError(f"Failed to download task result: {e}") from e

get_valid_params classmethod

get_valid_params(task_type: str) -> set[str]

Get valid parameters for a specific task type.

Parameters:

Name Type Description Default
task_type str

Harvest task type; see :attr:VALID_TASK_TYPES

required

Returns:

Type Description
set[str]

Set of valid parameter names for the task type

Raises:

Type Description
ValidationError

If task_type is invalid

Examples:

>>> asset_params = HarvesterAPI.get_valid_params("asset")
>>> print("Valid asset parameters:")
>>> for param in sorted(asset_params):
...     print(f"  - {param}")
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.UTILITY)
@classmethod
def get_valid_params(cls, task_type: str) -> set[str]:
    """Get valid parameters for a specific task type.

    Args:
        task_type: Harvest task type; see :attr:`VALID_TASK_TYPES`

    Returns:
        Set of valid parameter names for the task type

    Raises:
        ValidationError: If task_type is invalid

    Examples:
        >>> asset_params = HarvesterAPI.get_valid_params("asset")
        >>> print("Valid asset parameters:")
        >>> for param in sorted(asset_params):
        ...     print(f"  - {param}")
    """
    if task_type in WGET_PARAMS_TASK_TYPES:
        return cls.VALID_ASSET_PARAMS.copy()
    elif task_type == "visual":
        return cls.VALID_VISUAL_PARAMS.copy()
    elif task_type == "video":
        return cls.VALID_VIDEO_PARAMS.copy()
    else:
        raise ValidationError(
            f"Invalid task_type: {task_type}. "
            f"Valid types are: {', '.join(sorted(cls.VALID_TASK_TYPES))}"
        )

get_valid_task_types classmethod

get_valid_task_types() -> set[str]

Get all valid task types.

Returns:

Type Description
set[str]

Set of valid task type names

Examples:

>>> task_types = HarvesterAPI.get_valid_task_types()
>>> print("Supported task types:")
>>> for task_type in sorted(task_types):
...     print(f"  - {task_type}")
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(MethodType.UTILITY)
@classmethod
def get_valid_task_types(cls) -> set[str]:
    """Get all valid task types.

    Returns:
        Set of valid task type names

    Examples:
        >>> task_types = HarvesterAPI.get_valid_task_types()
        >>> print("Supported task types:")
        >>> for task_type in sorted(task_types):
        ...     print(f"  - {task_type}")
    """
    return set(cls.VALID_TASK_TYPES)

create_harvest_task_async async

create_harvest_task_async(task_type: str, urls: list[str], egress_info: dict[str, Any], dest_path: str = '/', dest_name: str | None = None, dest_bucket_id: str | None = None, task_params: dict[str, Any] | None = None, max_retries: int = 5, backoff_factor: int = 2, run_after: int | float | str | None = None, validate_egress: bool | None = None) -> str

Asynchronously create a new harvest task for web content collection.

This async version allows for non-blocking task creation and can be used in concurrent workflows for creating multiple tasks simultaneously.

Parameters:

Name Type Description Default
task_type str

Type of harvest task ("asset", "single", "visual", "video", or "wget")

required
urls list[str]

Array containing exactly one URL string to harvest

required
egress_info dict[str, Any]

Egress location information containing: - name: Egress location name (required, e.g., "New York, NY"). "asset" and "wget" tasks draw from the six asset datacenter locations (Singapore, Dubai, Frankfurt, Sao Paulo, Johannesburg, "New York, NY"): pass one of those or "World". Any other name logs a warning rather than raising. - categories: Optional object with connectivity, availability, protocol Each category field can be a string value or null. For non-"asset" task types, connectivity/protocol are validated against what is actually available at name.

required
dest_path str

Destination path in storage (defaults to root "/")

'/'
dest_name str | None

Name for the result file (auto-generated if not provided)

None
dest_bucket_id str | None

Destination bucket ID (uses default if not provided)

None
task_params dict[str, Any] | None

Dictionary of task-specific parameters

None
max_retries int

Maximum number of retries for explicit HTTP 429 rate-limit responses (default: 5). Ambiguous transport failures are not retried because task creation is non-idempotent.

5
backoff_factor int

Positive integer exponential backoff multiplier for rate-limit retries (default: 2)

2
run_after int | float | str | None

Optional schedule time — Unix epoch (int or float) or timezone-aware ISO 8601 string (e.g., "2026-02-13T18:30:00Z" or "2026-02-13T13:30:00-05:00"). Booleans and timezone-less strings are refused. When provided the task enters "wait" status until the specified time.

None
validate_egress bool | None

Look the egress up in its pool's live directory (:meth:check_harvest_egress) and log a warning if it is not listed. Never blocks the task. None (default) uses the LIVE_EGRESS_VALIDATION config key.

None

Returns:

Type Description
str

Task ID string for monitoring and managing the harvest task

Raises:

Type Description
HarvesterAPIError

If task creation fails

ValidationError

If parameters are invalid

Examples:

>>> import asyncio
>>> async def create_tasks():
...     harvester = HarvesterAPI(config)
...     task_id = await harvester.create_harvest_task_async(
...         task_type="asset",
...         urls=["https://example.com/files/"],
...         egress_info={"name": "New York, NY"},
...         dest_path="/downloads/"
...     )
...     return task_id
>>> task_id = asyncio.run(create_tasks())
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(
    MethodType.ASYNC_VARIANT,
    api_command="create_harvest_task",
    sync_of="create_harvest_task",
)
async def create_harvest_task_async(
    self,
    task_type: str,
    urls: list[str],
    egress_info: dict[str, Any],
    dest_path: str = "/",
    dest_name: str | None = None,
    dest_bucket_id: str | None = None,
    task_params: dict[str, Any] | None = None,
    max_retries: int = 5,
    backoff_factor: int = 2,
    run_after: int | float | str | None = None,
    validate_egress: bool | None = None,
) -> str:
    """Asynchronously create a new harvest task for web content collection.

    This async version allows for non-blocking task creation and can be used
    in concurrent workflows for creating multiple tasks simultaneously.

    Args:
        task_type: Type of harvest task ("asset", "single", "visual",
            "video", or "wget")
        urls: Array containing exactly one URL string to harvest
        egress_info: Egress location information containing:
            - name: Egress location name (required, e.g., "New York, NY").
              "asset" and "wget" tasks draw from the six asset datacenter
              locations (Singapore, Dubai, Frankfurt, Sao Paulo,
              Johannesburg, "New York, NY"): pass one of those or
              "World". Any other name logs a warning rather than raising.
            - categories: Optional object with connectivity, availability, protocol
              Each category field can be a string value or null. For
              non-"asset" task types, connectivity/protocol are validated
              against what is actually available at name.
        dest_path: Destination path in storage (defaults to root "/")
        dest_name: Name for the result file (auto-generated if not provided)
        dest_bucket_id: Destination bucket ID (uses default if not provided)
        task_params: Dictionary of task-specific parameters
        max_retries: Maximum number of retries for explicit HTTP 429
            rate-limit responses (default: 5). Ambiguous transport failures
            are not retried because task creation is non-idempotent.
        backoff_factor: Positive integer exponential backoff multiplier for
            rate-limit retries (default: 2)
        run_after: Optional schedule time — Unix epoch (int or float) or
            timezone-aware ISO 8601 string (e.g., "2026-02-13T18:30:00Z"
            or "2026-02-13T13:30:00-05:00"). Booleans and timezone-less
            strings are refused. When provided the task
            enters "wait" status until the specified time.
        validate_egress: Look the egress up in its pool's live directory
            (:meth:`check_harvest_egress`) and log a warning if it is not
            listed. Never blocks the task. ``None`` (default) uses the
            ``LIVE_EGRESS_VALIDATION`` config key.

    Returns:
        Task ID string for monitoring and managing the harvest task

    Raises:
        HarvesterAPIError: If task creation fails
        ValidationError: If parameters are invalid

    Examples:
        >>> import asyncio
        >>> async def create_tasks():
        ...     harvester = HarvesterAPI(config)
        ...     task_id = await harvester.create_harvest_task_async(
        ...         task_type="asset",
        ...         urls=["https://example.com/files/"],
        ...         egress_info={"name": "New York, NY"},
        ...         dest_path="/downloads/"
        ...     )
        ...     return task_id
        >>> task_id = asyncio.run(create_tasks())
    """
    validate_positive_int(max_retries, "max_retries")
    validate_positive_int(backoff_factor, "backoff_factor")

    task_params = self._normalize_task_params(task_params)
    top_level_keys = self._top_level_keys_for_type(task_type)
    validate_task_creation_params(
        task_type,
        urls,
        egress_info,
        dest_path,
        task_params,
        self.VALID_TASK_TYPES,
        self._valid_params_map,
        top_level_keys=top_level_keys,
    )

    if run_after is not None:
        validate_run_after(run_after)
    if self.live_egress_validation if validate_egress is None else validate_egress:
        # The lookup is a blocking request; keep it off the event loop.
        await asyncio.to_thread(self.check_harvest_egress, task_type, egress_info)

    normalized_egress_info = normalize_egress_info(egress_info)
    bucket_id = dest_bucket_id or self.dest_bucket_id
    final_dest_name = generate_dest_name_if_needed(dest_name, task_type)

    payload_commands = build_task_creation_payload(
        task_type,
        urls,
        normalized_egress_info,
        bucket_id,
        dest_path,
        final_dest_name,
        task_params or {},
        self.file_api.auth_token,
        run_after=run_after,
        top_level_keys=top_level_keys,
    )

    # Build API payload
    api_url = f"{self.base_url.rstrip('/')}/api/"
    payload = [{"command": "setauth", "data": self.auth_token}] + payload_commands

    async with httpx.AsyncClient(verify=self._ssl_context) as client:
        for attempt in range(max_retries):
            try:
                response = await client.post(
                    api_url,
                    json=payload,
                    headers={
                        "Accept": "application/json",
                        "Content-Type": "application/json",
                    },
                    timeout=httpx.Timeout(30),
                )
                if response.status_code == 200:
                    task_response = response.json()
                    # A rejected command still returns HTTP 200, with the
                    # error in place of (or nested inside) the result.
                    # Without this the server's message was lost behind
                    # "Unexpected response format: [...]".
                    try:
                        result_inner = self._extract_api_result(task_response)
                    except SiloError as e:
                        raise HarvesterAPIError(
                            f"Failed to create async harvest task: {e.message}",
                            response_data=e.response_data,
                        ) from e
                    result_task_id: str | None = (
                        result_inner.get("task_id")
                        if isinstance(result_inner, dict)
                        else None
                    )
                    if result_task_id:
                        self.logger.info(
                            "Created async harvest task: %s", result_task_id
                        )
                        return result_task_id

                    raise HarvesterAPIError(
                        "Unexpected response format: "
                        f"{self._format_response_for_log(task_response)}"
                    )

                elif response.status_code == 429:
                    if attempt == max_retries - 1:
                        break
                    try:
                        retry_after = int(
                            response.headers.get(
                                "Retry-After", backoff_factor**attempt
                            )
                        )
                    except ValueError:
                        retry_after = int(backoff_factor**attempt)
                    self.logger.warning(
                        "Rate limit hit, retrying after %s seconds...", retry_after
                    )
                    await asyncio.sleep(retry_after)
                else:
                    raise HarvesterAPIError(
                        f"API request failed with status code {response.status_code}"
                    )

            except httpx.HTTPError as e:
                raise HarvesterAPIError(
                    "Harvest task creation failed after the request may have "
                    "reached the server; the task may have been created. "
                    "The request was not retried to avoid creating a duplicate: "
                    f"{e}"
                ) from e
            except HarvesterAPIError:
                raise
            except Exception as e:
                raise HarvesterAPIError(
                    f"Failed to create async harvest task: {e}"
                ) from e

    raise HarvesterAPIError("Rate-limit retries exhausted; task was not created")

wait_for_completion_async async

wait_for_completion_async(task_id: str, max_wait_time: int = 1800, check_interval: int = 10, progress_callback: Callable[..., Any] | None = None) -> dict[str, Any]

Asynchronously wait for a harvest task to complete.

This async version allows for non-blocking task monitoring and can be used to monitor multiple tasks concurrently.

Parameters:

Name Type Description Default
task_id str

The ID of the task to monitor

required
max_wait_time int

Maximum time to wait in seconds (default: 30 minutes). A deadline: the last status check happens at it, not one check_interval past it.

1800
check_interval int

Time between status checks in seconds (default: 10)

10
progress_callback Callable[..., Any] | None

Optional function to call with progress updates

None

Returns:

Name Type Description
dict[str, Any]

Final task status dictionary when completed. Shape matches the

dict[str, Any]

synchronous wait_for_completion(): the raw server task dict

dict[str, Any]

merged with the normalized fields parse_task_response()

dict[str, Any]

produces. No server-reported field is dropped.

dict[str, Any]

Server-reported, passed through unchanged. These carry the same

dict[str, Any]

meanings as in find_harvest_task(), which documents each one

dict[str, Any]

in full:

dict[str, Any]
  • task_id: Task identifier
dict[str, Any]
  • status: One of wait, claimed, done or error. Always done on a successful return here, since an error status raises instead
dict[str, Any]
  • finished: True once the task has been resolved
dict[str, Any]
  • attempt_count: Attempts made, counting the last one
dict[str, Any]
  • last_error: Failure message, null on the success path
dict[str, Any]
  • hostname: Worker that ran the task
dict[str, Any]
  • org_id: Organization that owns the task
dict[str, Any]
  • create_ts: Creation time, a Unix epoch float
dict[str, Any]
  • start_ts: Claim time of the most recent attempt
dict[str, Any]
  • end_ts: Resolution time
dict[str, Any]
  • run_after: Earliest time the task was allowed to be claimed
dict[str, Any]
  • request_type: Task type, stripped and lower-cased
dict[str, Any]
  • target: Worker pool the task was routed to
dict[str, Any]
  • name: Caller-supplied label
dict[str, Any]
  • task_params: Parameters the task was created with
dict[str, Any]
  • dest_bucket_id: Storage bucket holding the results
dict[str, Any]
  • dest_path: Path within that bucket
dict[str, Any]
  • dest_name: Name of the result object
dict[str, Any]
  • egress_info: Egress selection in effect
dict[str, Any]
  • egress_tag: Superseded egress selector, null in practice
dict[str, Any]
  • result_file_id: File ID of the result
dict[str, Any]
  • task_results: Raw result payload from the worker
dict[str, Any]

Added on top by parse_task_response(). These are aliases, so

dict[str, Any]

each one duplicates a field above under a different name rather

dict[str, Any]

than replacing it:

dict[str, Any]
  • created_ts: Alias of create_ts
dict[str, Any]
  • started_ts: Alias of start_ts
dict[str, Any]
  • completed_ts: Alias of end_ts
dict[str, Any]
  • task_metadata: task_results["task-metadata"], or an empty dict when the worker recorded none
Note dict[str, Any]

find_harvest_task() returns the raw record without these four aliases. Code that polls with one and finishes with the other must read create_ts from the poll result and can read either name from this one.

Raises:

Type Description
HarvesterAPIError

If task fails or times out

ValidationError

If parameters are invalid

Examples:

>>> import asyncio
>>> async def monitor_task():
...     harvester = HarvesterAPI(config)
...     def progress_handler(task_info):
...         print(f"Status: {task_info['status']}")
...
...     result = await harvester.wait_for_completion_async(
...         task_id="task_123",
...         progress_callback=progress_handler
...     )
...     return result
>>> result = asyncio.run(monitor_task())
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(
    MethodType.ASYNC_VARIANT,
    api_command="find_harvest_task",
    sync_of="wait_for_completion",
)
async def wait_for_completion_async(
    self,
    task_id: str,
    max_wait_time: int = 1800,  # 30 minutes
    check_interval: int = 10,
    progress_callback: Callable[..., Any] | None = None,
) -> dict[str, Any]:
    """Asynchronously wait for a harvest task to complete.

    This async version allows for non-blocking task monitoring and can be used
    to monitor multiple tasks concurrently.

    Args:
        task_id: The ID of the task to monitor
        max_wait_time: Maximum time to wait in seconds (default: 30 minutes).
            A deadline: the last status check happens at it, not one
            ``check_interval`` past it.
        check_interval: Time between status checks in seconds (default: 10)
        progress_callback: Optional function to call with progress updates

    Returns:
        Final task status dictionary when completed. Shape matches the
        synchronous ``wait_for_completion()``: the raw server task dict
        merged with the normalized fields ``parse_task_response()``
        produces. No server-reported field is dropped.

        Server-reported, passed through unchanged. These carry the same
        meanings as in ``find_harvest_task()``, which documents each one
        in full:

        - task_id: Task identifier
        - status: One of ``wait``, ``claimed``, ``done`` or ``error``.
            Always ``done`` on a successful return here, since an ``error``
            status raises instead
        - finished: True once the task has been resolved
        - attempt_count: Attempts made, counting the last one
        - last_error: Failure message, null on the success path
        - hostname: Worker that ran the task
        - org_id: Organization that owns the task
        - create_ts: Creation time, a Unix epoch float
        - start_ts: Claim time of the most recent attempt
        - end_ts: Resolution time
        - run_after: Earliest time the task was allowed to be claimed
        - request_type: Task type, stripped and lower-cased
        - target: Worker pool the task was routed to
        - name: Caller-supplied label
        - task_params: Parameters the task was created with
        - dest_bucket_id: Storage bucket holding the results
        - dest_path: Path within that bucket
        - dest_name: Name of the result object
        - egress_info: Egress selection in effect
        - egress_tag: Superseded egress selector, null in practice
        - result_file_id: File ID of the result
        - task_results: Raw result payload from the worker

        Added on top by ``parse_task_response()``. These are aliases, so
        each one duplicates a field above under a different name rather
        than replacing it:

        - created_ts: Alias of ``create_ts``
        - started_ts: Alias of ``start_ts``
        - completed_ts: Alias of ``end_ts``
        - task_metadata: ``task_results["task-metadata"]``, or an empty
            dict when the worker recorded none

        Note:
            ``find_harvest_task()`` returns the raw record **without**
            these four aliases. Code that polls with one and finishes
            with the other must read ``create_ts`` from the poll result
            and can read either name from this one.

    Raises:
        HarvesterAPIError: If task fails or times out
        ValidationError: If parameters are invalid

    Examples:
        >>> import asyncio
        >>> async def monitor_task():
        ...     harvester = HarvesterAPI(config)
        ...     def progress_handler(task_info):
        ...         print(f"Status: {task_info['status']}")
        ...
        ...     result = await harvester.wait_for_completion_async(
        ...         task_id="task_123",
        ...         progress_callback=progress_handler
        ...     )
        ...     return result
        >>> result = asyncio.run(monitor_task())
    """
    self._require_task_id(task_id)

    if not isinstance(max_wait_time, int) or max_wait_time <= 0:
        raise ValidationError("max_wait_time must be a positive integer")

    if not isinstance(check_interval, int) or check_interval <= 0:
        raise ValidationError("check_interval must be a positive integer")

    # At least one check: an interval longer than the wait used to mean
    # none, so the call timed out without ever asking for the status.
    max_retries = max(1, max_wait_time // check_interval)
    api_url = f"{self.base_url.rstrip('/')}/api/"

    self.logger.info("Starting async status check loop for task ID: %s", task_id)
    retries = 0
    last_response = None
    start_time = time.time()

    try:
        async with httpx.AsyncClient(verify=self._ssl_context) as client:
            while retries < max_retries:
                # Bound the loop by wall-clock elapsed time in addition to
                # retry count. Retry count alone assumes each iteration
                # costs ~check_interval seconds, but a slow API response
                # (up to the 30s per-request timeout below) isn't counted
                # against max_wait_time, which lets the real wait exceed
                # it. This check only ever exits *earlier* than the retry
                # count would (never later), so it doesn't change
                # behavior when checks are fast.
                elapsed_time = time.time() - start_time
                if elapsed_time > max_wait_time:
                    self.logger.warning(
                        "Wall-clock max_wait_time (%s seconds) exceeded for "
                        "task %s before the retry budget was exhausted.",
                        max_wait_time,
                        task_id,
                    )
                    break

                self.logger.debug(
                    f"Checking status of task {task_id} "
                    f"(attempt {retries + 1}/{max_retries})"
                )

                payload = [
                    {"command": "setauth", "data": self.auth_token},
                    {"command": "find_harvest_task", "task_id": task_id},
                ]

                try:
                    response = await client.post(
                        api_url,
                        json=payload,
                        headers={
                            "Accept": "application/json",
                            "Content-Type": "application/json",
                        },
                        timeout=httpx.Timeout(30),
                    )
                    response.raise_for_status()
                    last_response = response.json()

                    # Both error shapes ({"error": ...} and a nested
                    # {"result": {"error": ...}}) raise here, as the sync
                    # find_harvest_task does. Reading result directly
                    # turned the first into [] and polled until timeout,
                    # and hit task_result[0] on the second (KeyError).
                    try:
                        task_result = self._extract_api_result(last_response)
                    except SiloError as e:
                        if e.response_data is None:
                            self.logger.warning(
                                "Invalid response format for task %s", task_id
                            )
                            raise HarvesterAPIError(
                                f"Invalid response format for task {task_id}"
                            ) from e
                        raise HarvesterAPIError(
                            f"Failed to find harvest task {task_id}: {e.message}",
                            response_data=e.response_data,
                        ) from e
                    if task_result and not (
                        isinstance(task_result, list)
                        and isinstance(task_result[0], dict)
                    ):
                        raise HarvesterAPIError(
                            f"Unexpected task result for task {task_id}: "
                            f"{self._format_response_for_log(last_response)}"
                        )

                    if task_result:
                        task_status = task_result[0].get("status", "")
                        self.logger.info(f"Task {task_id} status: {task_status}")

                        # Call progress callback if provided
                        if progress_callback is not None:
                            try:
                                progress_callback(task_result[0])
                            except Exception as e:
                                self.logger.warning(
                                    "Progress callback failed: %s", e
                                )

                        if task_status == "done":
                            self.logger.info(
                                f"Task {task_id} completed successfully."
                            )
                            task_info = parse_task_response(last_response)
                            if not task_info:
                                raise HarvesterAPIError(
                                    f"Failed to parse task response for {task_id}"
                                )
                            # Align this method's return shape with the
                            # synchronous wait_for_completion(), which
                            # returns the full raw server task dict (see
                            # find_harvest_task/task_result[0] above).
                            # parse_task_response() only surfaces 9
                            # hardcoded keys and drops the rest, including
                            # task_params, attempt_count, hostname,
                            # finished, target and task_results — merge the
                            # raw dict as the base and layer the normalized
                            # fields on top so nothing the server sent is
                            # lost. (This list previously cited "progress"
                            # and "worker_id"; the platform sends neither,
                            # so the justification for the merge rested on
                            # two fields that do not exist.)
                            full_task_info: dict[str, Any] = {
                                **task_result[0],
                                **task_info,
                            }
                            return full_task_info
                        elif task_status == "error":
                            error_msg = task_result[0].get(
                                "last_error", "Unknown error"
                            )
                            raise HarvesterAPIError(
                                f"Task {task_id} failed with server error: {error_msg}"
                            )
                        elif task_status == "wait":
                            run_after = task_result[0].get("run_after")
                            if run_after:
                                run_after_time = datetime.fromtimestamp(
                                    run_after, tz=UTC
                                )
                                self.logger.info(
                                    f"Task {task_id} is in 'wait' status, "
                                    f"scheduled to run at: {run_after_time} UTC."
                                )
                            else:
                                self.logger.info(
                                    f"Task {task_id} is in 'wait' status, "
                                    f"but no scheduled time is available."
                                )
                        elif task_status in ACTIVE_TASK_STATUSES:
                            self.logger.info(
                                f"Task {task_id} is in progress... "
                                f"Status: {task_status}"
                            )
                        else:
                            self.logger.warning(
                                f"Unexpected task status for task {task_id}: "
                                f"{task_status}"
                            )
                    else:
                        self.logger.warning(
                            f"No task result found for task {task_id}"
                        )

                except httpx.HTTPError as e:
                    self.logger.error(
                        f"Error: Request failed for task {task_id} with exception {e}."
                    )
                    raise HarvesterAPIError(
                        f"Request failed for task {task_id}: {e}"
                    ) from e

                delay = _poll_delay(check_interval, max_wait_time, elapsed_time)
                if delay is None:
                    break
                await asyncio.sleep(delay)
                retries += 1

        # Either the retry budget or the wall-clock max_wait_time was
        # exhausted without the task reaching a terminal state.
        self.logger.error(
            f"Timed out waiting for task {task_id}. "
            f"Task may not have completed successfully."
        )
        raise HarvesterAPIError(
            f"Task {task_id} did not complete within {max_wait_time} seconds"
        )

    except HarvesterAPIError:
        raise
    except Exception as e:
        raise HarvesterAPIError(f"Failed to monitor async task: {e}") from e

process_harvest_workflow_async async

process_harvest_workflow_async(task_type: str, urls: list[str], egress_info: dict[str, Any], output_dir: str, dest_path: str = '/', dest_name: str | None = None, task_params: dict[str, Any] | None = None, use_chunked_download: bool = False, max_wait_time: int = 1800, check_interval: int = 20) -> dict[str, Any] | None

Process a complete harvest workflow asynchronously.

This high-level async method orchestrates the entire harvest workflow: 1. Creates a harvest task 2. Monitors its status until completion 3. Downloads the result file 4. Returns comprehensive task information

Parameters:

Name Type Description Default
task_type str

Type of harvest task ("asset", "single", "visual", "video", or "wget")

required
urls list[str]

Array containing exactly one URL string to harvest

required
egress_info dict[str, Any]

Egress location information containing: - name: Egress location name (required, e.g., "New York, NY"). "asset" and "wget" tasks draw from the six asset datacenter locations (Singapore, Dubai, Frankfurt, Sao Paulo, Johannesburg, "New York, NY"): pass one of those or "World". Any other name logs a warning rather than raising. - categories: Optional object with connectivity, availability, protocol Each category field can be a string value or null. For non-"asset" task types, connectivity/protocol are validated against what is actually available at name.

required
output_dir str

Directory to save downloaded results

required
dest_path str

Destination path in storage (default: "/")

'/'
dest_name str | None

Name for result file (auto-generated if not provided)

None
task_params dict[str, Any] | None

Dictionary of task-specific parameters

None
use_chunked_download bool

Whether to use chunked download (default: False)

False
max_wait_time int

Maximum time to wait for completion (default: 30 minutes)

1800
check_interval int

Time between status checks (default: 20 seconds)

20

Returns:

Type Description
dict[str, Any] | None

Dictionary with task information and results, None if failed

Raises:

Type Description
HarvesterAPIError

If workflow fails

ValidationError

If parameters are invalid

Examples:

>>> import asyncio
>>> async def harvest_site():
...     harvester = HarvesterAPI(config)
...     result = await harvester.process_harvest_workflow_async(
...         task_type="asset",
...         urls=["https://example.com/documents/"],
...         egress_info={"name": "New York, NY"},
...         output_dir="./downloads",
...         task_params={"recursive": True, "level": 2}
...     )
...     return result
>>> result = asyncio.run(harvest_site())
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(
    MethodType.ASYNC_CONVENIENCE,
    wraps=[
        "create_harvest_task_async",
        "wait_for_completion_async",
        "download_file_chunked_async",
        "download_file_with_retry_async",
    ],
)
async def process_harvest_workflow_async(
    self,
    task_type: str,
    urls: list[str],
    egress_info: dict[str, Any],
    output_dir: str,
    dest_path: str = "/",
    dest_name: str | None = None,
    task_params: dict[str, Any] | None = None,
    use_chunked_download: bool = False,
    max_wait_time: int = 1800,
    check_interval: int = 20,
) -> dict[str, Any] | None:
    """Process a complete harvest workflow asynchronously.

    This high-level async method orchestrates the entire harvest workflow:
    1. Creates a harvest task
    2. Monitors its status until completion
    3. Downloads the result file
    4. Returns comprehensive task information

    Args:
        task_type: Type of harvest task ("asset", "single", "visual",
            "video", or "wget")
        urls: Array containing exactly one URL string to harvest
        egress_info: Egress location information containing:
            - name: Egress location name (required, e.g., "New York, NY").
              "asset" and "wget" tasks draw from the six asset datacenter
              locations (Singapore, Dubai, Frankfurt, Sao Paulo,
              Johannesburg, "New York, NY"): pass one of those or
              "World". Any other name logs a warning rather than raising.
            - categories: Optional object with connectivity, availability, protocol
              Each category field can be a string value or null. For
              non-"asset" task types, connectivity/protocol are validated
              against what is actually available at name.
        output_dir: Directory to save downloaded results
        dest_path: Destination path in storage (default: "/")
        dest_name: Name for result file (auto-generated if not provided)
        task_params: Dictionary of task-specific parameters
        use_chunked_download: Whether to use chunked download (default: False)
        max_wait_time: Maximum time to wait for completion (default: 30 minutes)
        check_interval: Time between status checks (default: 20 seconds)

    Returns:
        Dictionary with task information and results, None if failed

    Raises:
        HarvesterAPIError: If workflow fails
        ValidationError: If parameters are invalid

    Examples:
        >>> import asyncio
        >>> async def harvest_site():
        ...     harvester = HarvesterAPI(config)
        ...     result = await harvester.process_harvest_workflow_async(
        ...         task_type="asset",
        ...         urls=["https://example.com/documents/"],
        ...         egress_info={"name": "New York, NY"},
        ...         output_dir="./downloads",
        ...         task_params={"recursive": True, "level": 2}
        ...     )
        ...     return result
        >>> result = asyncio.run(harvest_site())
    """
    # Validate egress_info with task-type-aware validation
    validate_egress_for_task_type(task_type, egress_info)

    # Create output directory
    output_path = Path(output_dir)
    await asyncio.to_thread(output_path.mkdir, parents=True, exist_ok=True)

    self.logger.info(
        "Processing %s task for URL: %s", task_type, urls[0] if urls else "<empty>"
    )

    try:
        # Step 1: Create harvest task
        task_id = await self.create_harvest_task_async(
            task_type=task_type,
            urls=urls,
            egress_info=egress_info,
            dest_path=dest_path,
            dest_name=dest_name,
            task_params=task_params,
        )

        self.logger.info("Started %s task with ID: %s", task_type, task_id)

        # Step 2: Monitor task status
        task_info = await self.wait_for_completion_async(
            task_id=task_id,
            max_wait_time=max_wait_time,
            check_interval=check_interval,
        )

        if not task_info:
            self.logger.error("No status response received for task %s", task_id)
            return None

        result_file_id = task_info.get("result_file_id")
        if not result_file_id:
            self.logger.error("Missing result file ID for task %s", task_id)
            return task_info

        # Step 3: Download result file
        output_file_path = output_path / f"harvest-{task_id}.zip"

        try:
            self.logger.info("Downloading result for task %s", task_id)

            if use_chunked_download:
                downloaded_file = await self.file_api.download_file_chunked_async(
                    file_id=result_file_id, output_path=str(output_file_path)
                )
            else:
                downloaded_file = (
                    await self.file_api.download_file_with_retry_async(
                        file_id=result_file_id, output_path=str(output_file_path)
                    )
                )

            if downloaded_file:
                self.logger.info("Downloaded file saved to: %s", downloaded_file)
                task_info["downloaded_file"] = str(downloaded_file)
                task_info["download_success"] = True
            else:
                self.logger.error(
                    "Failed to download result file for task %s", task_id
                )
                task_info["download_success"] = False

        except Exception as e:
            self.logger.error("Error downloading file for task %s: %s", task_id, e)
            task_info["download_success"] = False
            task_info["download_error"] = str(e)

        return task_info

    except Exception as e:
        if isinstance(e, (HarvesterAPIError, ValidationError)):
            raise
        raise HarvesterAPIError(
            f"Failed to process async harvest workflow: {e}"
        ) from e

bulk_create_and_monitor_async async

bulk_create_and_monitor_async(tasks: list[dict[str, Any]], output_dir: str, max_concurrent: int = 5, egress_info: dict[str, Any] | None = None) -> list[dict[str, Any] | None]

Create and monitor multiple harvest tasks concurrently. This method processes multiple harvest tasks in parallel with configurable concurrency limits to efficiently handle bulk harvesting operations.

Parameters:

Name Type Description Default
tasks list[dict[str, Any]]

List of task dictionaries, each containing: - task_type: Type of harvest task - urls: List containing exactly one URL string to harvest (canonical key — matches create_harvest_task()). A single url string key is also accepted as a convenience alias and is wrapped into a one-item list. - egress_info: Optional per-task egress location dict. When provided, it overrides the bulk-level egress_info for this task only — this is how you mix asset tasks (which need one of the 6 datacenter locations) with non-asset tasks in the same call. - params: Task-specific parameters (dict format) - dest_path: Optional destination path (default: "/") - dest_name: Optional destination name

required
output_dir str

Directory to save downloaded results

required
max_concurrent int

Positive maximum number of concurrent tasks (default: 5)

5
egress_info dict[str, Any] | None

Default egress location information used for any task that doesn't specify its own egress_info. When omitted, each task gets the default for its type: {"name": "New York, NY"} for "asset" and "wget", {"name": "World"} (any location) for every other type. Note: asset and wget tasks draw from the six asset datacenter locations (Singapore, Dubai, Frankfurt, Sao Paulo, Johannesburg, "New York, NY") or "World". Another name, such as "London", only logs a warning, and the task fails at run time unless an asset location sits beneath it. If you pass a bulk-level value and mix asset and non-asset tasks, set egress_info per-task (see above) or use separate calls.

None

Returns:

Type Description
list[dict[str, Any] | None]

List of task results (same order as input), None for failed tasks

Raises:

Type Description
HarvesterAPIError

If bulk processing fails

ValidationError

If parameters are invalid

Examples:

>>> import asyncio
>>> async def bulk_harvest():
...     harvester = HarvesterAPI(config)
...     tasks = [
...         {
...             "task_type": "asset",
...             "urls": ["https://site1.com"],
...             "egress_info": {"name": "New York, NY"},
...             "params": {"recursive": True}
...         },
...         {
...             "task_type": "visual",
...             "urls": ["https://site2.com"],
...             "params": {"output_pdf": True}
...         }
...     ]
...     results = await harvester.bulk_create_and_monitor_async(
...         tasks=tasks,
...         output_dir="./downloads",
...         max_concurrent=3
...     )
...     return results
>>> results = asyncio.run(bulk_harvest())
Source code in silo_sdk/harvesting/harvester_api.py
@api_tag(
    MethodType.ASYNC_CONVENIENCE,
    wraps=["process_harvest_workflow_async"],
)
async def bulk_create_and_monitor_async(
    self,
    tasks: list[dict[str, Any]],
    output_dir: str,
    max_concurrent: int = 5,
    egress_info: dict[str, Any] | None = None,
) -> list[dict[str, Any] | None]:
    """Create and monitor multiple harvest tasks concurrently.
    This method processes multiple harvest tasks in parallel with configurable
    concurrency limits to efficiently handle bulk harvesting operations.

    Args:
        tasks: List of task dictionaries, each containing:
            - task_type: Type of harvest task
            - urls: List containing exactly one URL string to harvest
              (canonical key — matches ``create_harvest_task()``).
              A single ``url`` string key is also accepted as a
              convenience alias and is wrapped into a one-item list.
            - egress_info: Optional per-task egress location dict. When
              provided, it overrides the bulk-level ``egress_info`` for
              this task only — this is how you mix asset tasks (which
              need one of the 6 datacenter locations) with non-asset
              tasks in the same call.
            - params: Task-specific parameters (dict format)
            - dest_path: Optional destination path (default: "/")
            - dest_name: Optional destination name
        output_dir: Directory to save downloaded results
        max_concurrent: Positive maximum number of concurrent tasks (default: 5)
        egress_info: Default egress location information used for any
            task that doesn't specify its own ``egress_info``. When
            omitted, each task gets the default for its type:
            ``{"name": "New York, NY"}`` for ``"asset"`` and ``"wget"``,
            ``{"name": "World"}`` (any location) for every other type.
            Note: asset and wget tasks draw from the six asset
            datacenter locations (Singapore, Dubai, Frankfurt, Sao Paulo,
            Johannesburg, ``"New York, NY"``) or ``"World"``. Another name,
            such as ``"London"``, only logs a warning, and the task fails
            at run time unless an asset location sits beneath it. If you
            pass a bulk-level value and mix asset and non-asset tasks, set
            ``egress_info`` per-task (see above) or use separate calls.

    Returns:
        List of task results (same order as input), None for failed tasks

    Raises:
        HarvesterAPIError: If bulk processing fails
        ValidationError: If parameters are invalid

    Examples:
        >>> import asyncio
        >>> async def bulk_harvest():
        ...     harvester = HarvesterAPI(config)
        ...     tasks = [
        ...         {
        ...             "task_type": "asset",
        ...             "urls": ["https://site1.com"],
        ...             "egress_info": {"name": "New York, NY"},
        ...             "params": {"recursive": True}
        ...         },
        ...         {
        ...             "task_type": "visual",
        ...             "urls": ["https://site2.com"],
        ...             "params": {"output_pdf": True}
        ...         }
        ...     ]
        ...     results = await harvester.bulk_create_and_monitor_async(
        ...         tasks=tasks,
        ...         output_dir="./downloads",
        ...         max_concurrent=3
        ...     )
        ...     return results
        >>> results = asyncio.run(bulk_harvest())
    """
    validate_positive_int(max_concurrent, "max_concurrent")

    # Create semaphore to limit concurrent tasks
    semaphore = asyncio.Semaphore(max_concurrent)

    async def process_single_task(
        task_config: dict[str, Any],
    ) -> dict[str, Any] | None:
        """Process a single task with semaphore control."""
        async with semaphore:
            try:
                # Get task_params from either 'params' or 'task_params' key
                task_params = task_config.get("task_params") or task_config.get(
                    "params", {}
                )

                # Accept both 'urls' (list — canonical, matches
                # create_harvest_task/create_harvest_task_async) and a
                # single 'url' string as a convenience alias, so callers
                # don't hit a KeyError from a docstring/param mismatch.
                task_urls = task_config.get("urls")
                if task_urls is None:
                    single_url = task_config.get("url")
                    task_urls = [single_url] if single_url is not None else None
                if not task_urls:
                    raise ValidationError(
                        "Each task dict must include a non-empty 'urls' "
                        "list (or a single 'url' string)"
                    )

                # Honor this task's own egress_info when provided,
                # falling back to the bulk-level default. Previously the
                # outer-scope egress_info was always used, making it
                # impossible to mix asset tasks (which require one of
                # the 6 datacenter locations) with non-asset tasks in a
                # single bulk call, despite the docstring advertising
                # per-task egress as the way to do so.
                #
                # With neither, use the default for this task's type: a
                # single bulk-wide default sent "New York, NY" to asset
                # tasks, which only accept the datacenter locations.
                task_egress_info = task_config.get("egress_info", egress_info)
                if task_egress_info is None:
                    task_egress_info = default_egress_for_task_type(
                        task_config["task_type"]
                    )

                return await self.process_harvest_workflow_async(
                    task_type=task_config["task_type"],
                    urls=task_urls,
                    egress_info=task_egress_info,
                    output_dir=output_dir,
                    dest_path=task_config.get("dest_path", "/"),
                    dest_name=task_config.get("dest_name"),
                    task_params=(
                        task_params if isinstance(task_params, dict) else None
                    ),
                )
            except Exception as e:
                self.logger.error("Task failed: %s", e)
                return None

    # Process all tasks concurrently
    self.logger.info(
        "Processing %d tasks with max concurrency: %d", len(tasks), max_concurrent
    )

    try:
        results = await asyncio.gather(
            *[process_single_task(task) for task in tasks]
        )

        processed_results: list[dict[str, Any] | None] = [
            r if isinstance(r, dict) else None for r in results
        ]

        successful_tasks = sum(1 for r in processed_results if r is not None)
        self.logger.info(
            f"Completed bulk processing: {successful_tasks}/{len(tasks)} tasks successful"
        )

        return processed_results

    except Exception as e:
        if isinstance(e, (HarvesterAPIError, ValidationError)):
            raise
        raise HarvesterAPIError(f"Failed to process bulk async harvest: {e}") from e

Task Type and Status Constants

silo_sdk.utils.harvester_utils defines the task-type and task-status groups the SDK's validators use. All are immutable frozensets.

Constant Contents
VALID_TASK_TYPES Every accepted task_type: asset, visual, video, single, wget. Also available as HarvesterAPI.VALID_TASK_TYPES; HarvesterAPI.get_valid_task_types() returns a mutable copy.
WGET_PARAMS_TASK_TYPES Types sent with wget_params, which share one parameter set and the same builder options: asset, single, wget.
ASSET_TASK_TYPES Types that follow asset's rules — the asset egress pool, no TOR, no all_files: asset, wget.
ACTIVE_TASK_STATUSES Statuses of a task a worker has picked up and is processing.
IN_PROGRESS_TASK_STATUSES ACTIVE_TASK_STATUSES plus wait, a task that is queued or scheduled (run_after) and not yet picked up.
from silo_sdk.utils.harvester_utils import ASSET_TASK_TYPES, VALID_TASK_TYPES

Egress by task type

Harvest tasks take egress from one of two pools, chosen by task type, for the organization the harvest token administers:

Task types Pool (service_type)
visual, video, single egress — the pool browsing contexts use
asset, wget harvest-asset

HarvesterAPI.list_harvest_egresses(task_type) asks the platform for that pool, for HARVEST_EGRESS_ORG, or TOP_ORG if that is unset (it needs ADMIN_TOKEN). With LIVE_EGRESS_VALIDATION or validate_egress=True, task creation checks the egress the same way and logs a warning if it is not listed. It never blocks the task: an organization's Launcher Options hide regions from the list but not from harvest, so the list can omit an egress a task may still use.

An org lookup sees only egress provisioned to that org and the orgs above it, so an organization whose egress is provisioned to its sub-orgs lists nothing for itself. An empty list therefore does not mean a harvest will fail: query the sub-org instead. Harvest resolves egress for the org the harvest token belongs to; when that is a sub-org of TOP_ORG (common when egress is provisioned to a workspace sub-org), set HARVEST_EGRESS_ORG (A8_HARVEST_EGRESS_ORG) to it, or pass the sub-org's org_id to list_harvest_egresses. Otherwise the lookup sees nothing and the check is skipped. An empty list is never checked against.