Callbacks and Error Handling¶

Callbacks drive the happy path, while error handlers keep failures contained. aioscraper routes both through the request lifecycle so you can react at the right moment.

Key points

Request.callback runs only on successful responses.
Request.errback handles HTTP statuses >=400 and unexpected exceptions from callbacks or middlewares.
Request.cb_kwargs are merged into callback/errback arguments alongside framework dependencies (send_request, pipeline, etc.).
Use @compiled decorator on callbacks for optimized dependency injection.

import logging

from aioscraper import AIOScraper, Request, Response, SendRequest, Pipeline
from aioscraper.exceptions import HTTPException

scraper = AIOScraper()


@scraper
async def scrape(send_request: SendRequest):
    await send_request(
        Request(
            url="https://example.com/api/article",
            callback=handle_response,
            errback=handle_error,
        )
    )


async def handle_response(response: Response, pipeline: Pipeline):
    # process data
    ...


async def handle_error(exc: Exception, request: Request):
    if isinstance(exc, HTTPException):
        logging.warning("HTTP %s for %s", exc.status_code, request.url)
    else:
        logging.exception("Unhandled error for %s", request.url)

Optimizing callbacks¶

By default, aioscraper uses runtime inspection to inject dependencies into callbacks. For performance-critical scrapers, use the @compiled decorator to pre-compute parameter filtering at import time.

from aioscraper import AIOScraper, Request, Response, SendRequest, compiled

scraper = AIOScraper()


@scraper
async def scrape(send_request: SendRequest):
    await send_request(Request(url="https://api.example.com/data", callback=parse))


@compiled
async def parse(response: Response, send_request: SendRequest):
    # Dependencies injected with zero runtime overhead
    data = await response.json()
    # process data...

The @compiled decorator:

Caches function parameters at import time instead of inspecting on every call
Eliminates repeated inspect.signature() calls from the hot path
Provides ~10-30% performance improvement for callback execution
Works with both callbacks and errbacks
Maintains full compatibility with dependency injection

class aioscraper.types.session.Request(*, url, method=<HTTPMethod.GET>, params=None, data=None, json_data=None, files=None, cookies=None, headers=None, auth=None, proxy=None, proxy_auth=None, proxy_headers=None, timeout=None, allow_redirects=True, max_redirects=10, delay=None, priority=0, callback=None, cb_kwargs=<factory>, errback=None, state=<factory>)[source]

Bases: object

Represents an HTTP request with all its parameters.

Parameters:

url (str) – Target URL
method (str) – HTTP method
params (QueryParams | None) – URL query parameters
data (Any) – Request body data
files (RequestFiles | None) – Multipart files mapping
json_data (Any) – JSON data to be sent in the request body
cookies (RequestCookies | None) – Request cookies
headers (RequestHeaders | None) – Request headers
auth (BasicAuth | None) – Basic authentication credentials
proxy (str | None) – Proxy URL (per-request proxies are honored only by the aiohttp backend)
proxy_auth (BasicAuth | None) – Proxy authentication credentials
proxy_headers (RequestHeaders | None) – Proxy headers
timeout (float | None) – Request timeout in seconds
allow_redirects (bool) – Whether to follow HTTP redirects
max_redirects (int) – Maximum number of redirects to follow
delay (float | None) – Delay before sending the request
priority (int) – Priority of the request
callback (Callable[..., Awaitable] | None) – Async callback function to be called after successful request
cb_kwargs (dict[str, Any]) – Keyword arguments for the callback function
errback (Callable[..., Awaitable] | None) – Async error callback function
state (dict[str, Any]) – State for middlewares

aioscraper.compiled(func)[source]

Decorator that optimizes dependency injection by caching function parameters.

Replaces runtime inspection with compile-time parameter extraction.

Parameters:: func (Callable[[...], Any])
Return type:: Callable[[…], Any]