diff --git a/README.md b/README.md index 8d27205..ede8079 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,27 @@ -# website-crawler -Multi-Site Keyword Crawler with Fulltext & Port Scanning +# 🕷️ Multi-Site Web Crawler with Fulltext & Port Scanning + +A powerful Python-based crawler that performs keyword discovery, fulltext extraction, and optional per-host port scanning — across multiple domains at once. + +## 🚀 Features + +- 🔍 **Multi-Site, Multi-Keyword Search** + Crawl multiple starting URLs and search for multiple terms in parallel. + +- 🧠 **Fulltext Capture** + Stores all visible page text for later NLP or forensic analysis. + +- 📊 **CSV Export** + All metadata — titles, URLs, IPs, keywords, fulltext, open ports — is exported to `.csv`. + +- 🔌 **Optional Custom Port Scan** + Scan user-defined ports on matched IP addresses to discover open services. + +- 🖥️ **GUI & CLI Interface** + - Use `crawler_gui.py` with a browser interface (Streamlit) + - Use `crawler_cli.py` for headless jobs or automation + +- 🐳 **Docker & Portainer Ready** + Ships with a `Dockerfile` and `docker-compose.yml` — deployable to cloud, local or container orchestration platforms. + + +