CLI¶
Run scrapers from the command line without wiring up the event loop yourself.
pip install "aioscraper[aiohttp]"
aioscraper scraper
See the minimal code in Quickstart.
Entrypoint contract¶
The CLI loads a module (file path or module.path) and optionally a specific attribute using module:attr.
Entry rules:
Without
:attr: the CLI looks for ascraperattribute that is either anAIOScraperinstance or a callable returning one.With
:attrpointing to anAIOScraper: the CLI uses that instance.With
:attrpointing to a callable (sync or async): the CLI executes/awaits it and expects anAIOScraperinstance in return.
Examples¶
aioscraper scraper # uses scraper variable from scraper.py
aioscraper mypkg.scraper:custom_app # uses custom_app AIOScraper instance
aioscraper mypkg.factory:make # calls make() (sync factory)
aioscraper mypkg.factory:make_async # awaits make_async() (async factory)
For resource setup/teardown around the same scraper instance, attach a lifespan(scraper) when constructing the scraper in code (see Lifespan).
Instance or factory¶
The two forms differ in when the scraper is built. A module-level instance is built while the CLI imports the module, so its configuration is resolved at import time; a factory runs afterwards, once the arguments are parsed.
That is what makes the factory the place for a configuration loader of your own:
from aioscraper import AIOScraper
def make() -> AIOScraper:
return AIOScraper(scrape, config=load_my_config())
examples/third_party_config.py is a runnable version.
Running without the CLI¶
You can run the same scraper programmatically using run_scraper:
import asyncio
from aioscraper import AIOScraper, Request, ScheduleRequest, run_scraper
from aioscraper.config import load_config
async def scrape(schedule_request: ScheduleRequest):
await schedule_request(Request(url="https://example.com"))
async def main() -> int:
scraper = AIOScraper(scrape, config=load_config())
result = await run_scraper(scraper)
if result.interrupted:
return 130
return 0 if result.ok else 1
if __name__ == "__main__":
raise SystemExit(asyncio.run(main()))
This gives you the same signal handling and graceful shutdown behavior as the CLI.
run_scraper expects scraper.config to be set ahead of time, which is why the example passes config=load_config() to the constructor.
run_scraper returns a RunResult and never acts on it: turning the outcome into an exit code is the caller’s decision, and the CLI is one implementation of it. The handlers turn SIGINT/SIGTERM into an event, so KeyboardInterrupt never reaches the caller — result.interrupted is how a signaled run is told apart from a clean one.
Overlapping runs share the handlers and each gets the signal; the last one to end restores the previous handlers, except one registered with loop.add_signal_handler, which asyncio does not expose. An application that handles signals itself passes install_signal_handlers=False.
Configuration¶
An AIOScraper built without config reads it from environment variables, falling back to Config defaults; one built with config reads no environment variables.
CLI flags are applied on top of scraper.config either way, so they override both.
See Configuration for detailed configuration options and examples.
CLI flags¶
--concurrent-requests: Max concurrent requests (overridesSCHEDULER_CONCURRENT_REQUESTS).--pending-requests: Pending requests to keep queued (overridesSCHEDULER_PENDING_REQUESTS).--allow-partial-success: Exit0even when the run recorded unhandled errors (overridesEXECUTION_ON_ERROR, see unhandled errors).
Exit codes¶
0- the run finished, with nothing recorded or with errors the policy waives.1- the run finished but recorded unhandled errors underErrorPolicy.FAIL(the default).124-execution.timeoutexpired, so the run never finished. Not waivable by the policy or the flag.130- SIGINT/SIGTERM stopped the run. Takes precedence over the rest.
Environment variables¶
All environment variables map directly to fields in Config and its nested configuration classes.
The CLI reads these variables automatically. For programmatic use, call load_config to read environment variables and construct a Config instance.
SessionConfig¶
HTTP session and client behavior.
SESSION_REQUEST_TIMEOUT→timeoutSESSION_SSL→sslSESSION_PROXY→proxy(docs)SESSION_TRUST_ENV→trust_env(docs)SESSION_HTTP_BACKEND→http_backendSESSION_MAX_RESPONSE_BODY_SIZE→max_response_body_size;0,noneorunlimitedturns the cap off (docs)SESSION_MAX_ERROR_BODY_SIZE→max_error_body_size(docs)SESSION_BUFFER_BODY→buffer_body(docs)
RequestRetryConfig¶
Retry behavior (docs).
SESSION_RETRY_ENABLED→enabledSESSION_RETRY_ATTEMPTS→attemptsSESSION_RETRY_BACKOFF→backoffSESSION_RETRY_BASE_DELAY→base_delaySESSION_RETRY_MAX_DELAY→max_delaySESSION_RETRY_MAX_RETRY_AFTER→max_retry_afterSESSION_RETRY_STATUSES→statusesSESSION_RETRY_EXCEPTIONS→exceptionsSESSION_RETRY_METHODS→methods
RateLimitConfig¶
Rate limiting behavior (docs).
SESSION_RATE_LIMIT_GROUP_CONCURRENCY needs SESSION_RATE_LIMIT_PER_GROUP=true as well, since
it limits one group at a time; without it load_config
raises ConfigValidationError.
SESSION_RATE_LIMIT_PER_GROUP→per_groupSESSION_RATE_LIMIT_INTERVAL→default_intervalSESSION_RATE_LIMIT_CLEANUP_TIMEOUT→cleanup_timeoutSESSION_RATE_LIMIT_GROUP_CONCURRENCY→group_concurrency
AdaptiveRateLimitConfig¶
Adaptive rate limiting (docs).
Set SESSION_RATE_LIMIT_ADAPTIVE_ENABLED=true to enable and configure other parameters. It needs
SESSION_RATE_LIMIT_PER_GROUP=true as well, since adaptive paces a group at a time; without it
load_config raises
ConfigValidationError.
SESSION_RATE_LIMIT_ADAPTIVE_MIN_INTERVAL→min_intervalSESSION_RATE_LIMIT_ADAPTIVE_MAX_INTERVAL→max_intervalSESSION_RATE_LIMIT_ADAPTIVE_INCREASE_FACTOR→increase_factorSESSION_RATE_LIMIT_ADAPTIVE_DECREASE_STEP→decrease_stepSESSION_RATE_LIMIT_ADAPTIVE_SUCCESS_THRESHOLD→success_thresholdSESSION_RATE_LIMIT_ADAPTIVE_EWMA_ALPHA→ewma_alphaSESSION_RATE_LIMIT_ADAPTIVE_RESPECT_RETRY_AFTER→respect_retry_afterSESSION_RATE_LIMIT_ADAPTIVE_INHERIT_RETRY_TRIGGERS→inherit_retry_triggers
SchedulerConfig¶
Request scheduler behavior.
SCHEDULER_CONCURRENT_REQUESTS→concurrent_requestsSCHEDULER_PENDING_REQUESTS→pending_requestsSCHEDULER_CLOSE_TIMEOUT→close_timeoutSCHEDULER_READY_QUEUE_MAX_SIZE→ready_queue_max_size(throttles the scraper entrypoint; not a hard cap)
ExecutionConfig¶
Execution and shutdown behavior.
EXECUTION_TIMEOUT→timeoutEXECUTION_SHUTDOWN_TIMEOUT→shutdown_timeoutEXECUTION_SHUTDOWN_CHECK_INTERVAL→shutdown_check_intervalEXECUTION_ON_ERROR→on_error(failorlog)EXECUTION_MAX_RETAINED_ERRORS→max_retained_errors(docs)EXECUTION_LOG_LEVEL→log_level
PipelineConfig¶
Pipeline dispatching behavior.
PIPELINE_STRICT→strict