28 lines
1020 B
Markdown
28 lines
1020 B
Markdown
# 🕷️ Multi-Site Web Crawler with Fulltext & Port Scanning
|
|
|
|
A powerful Python-based crawler that performs keyword discovery, fulltext extraction, and optional per-host port scanning — across multiple domains at once.
|
|
|
|
## 🚀 Features
|
|
|
|
- 🔍 **Multi-Site, Multi-Keyword Search**
|
|
Crawl multiple starting URLs and search for multiple terms in parallel.
|
|
|
|
- 🧠 **Fulltext Capture**
|
|
Stores all visible page text for later NLP or forensic analysis.
|
|
|
|
- 📊 **CSV Export**
|
|
All metadata — titles, URLs, IPs, keywords, fulltext, open ports — is exported to `.csv`.
|
|
|
|
- 🔌 **Optional Custom Port Scan**
|
|
Scan user-defined ports on matched IP addresses to discover open services.
|
|
|
|
- 🖥️ **GUI & CLI Interface**
|
|
- Use `crawler_gui.py` with a browser interface (Streamlit)
|
|
- Use `crawler_cli.py` for headless jobs or automation
|
|
|
|
- 🐳 **Docker & Portainer Ready**
|
|
Ships with a `Dockerfile` and `docker-compose.yml` — deployable to cloud, local or container orchestration platforms.
|
|
|
|
|
|
|