url2md - Web Scraping & Search API

A self-hosted web scraping and search API built for modern AI applications. Scrape any URL to clean LLM-ready Markdown, search the web via an integrated SearXNG instance, or perform both actions in a single unified API call.

Key Capabilities

URL Scraping (POST /v2/scrape): Converts complex HTML pages into clean Markdown, stripping advertisements, cookie banners, navigation menus, and non-content elements.
Web Search Integration (POST /v2/search): Queries SearXNG across multiple search engines with privacy preservation.
Combined Search & Scrape (POST /v2/search-and-scrape): Runs a web search and immediately scrapes the top resulting pages into LLM-ready context in a single request.
Interactive Documentation: Built-in Swagger/OpenAPI interactive documentation available at /docs.
Health Checks & Monitoring: Detailed system and engine health endpoints (/healthz, /health/deep).

Quick Start with Docker

Launch the complete stack (API + SearXNG) with Docker Compose:

# 1. Copy environment template
cp .env.example .env

# 2. Generate SearXNG secret key
echo "SEARXNG_SECRET_KEY=$(openssl rand -hex 32)" >> .env

# 3. Launch full stack
docker compose up -d --build

API Usage Example

curl -X POST http://localhost:3000/v2/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "onlyMainContent": true
  }'

License

Distributed under the MIT License.