A powerful Python-based crawler that performs keyword discovery, fulltext extraction, and optional per-host port scanning — across multiple domains at once.

🚀 Features

🔍 Multi-Site, Multi-Keyword Search
Crawl multiple starting URLs and search for multiple terms in parallel.
🧠 Fulltext Capture
Stores all visible page text for later NLP or forensic analysis.
📊 CSV Export
All metadata — titles, URLs, IPs, keywords, fulltext, open ports — is exported to .csv.
🔌 Optional Custom Port Scan
Scan user-defined ports on matched IP addresses to discover open services.
🖥️ GUI & CLI Interface
- Use crawler_gui.py with a browser interface (Streamlit)
- Use crawler_cli.py for headless jobs or automation
🐳 Docker & Portainer Ready
Ships with a Dockerfile and docker-compose.yml — deployable to cloud, local or container orchestration platforms.