From 06ba6eff6f369abd1479fa3bf6b9a1e8809d3a04 Mon Sep 17 00:00:00 2001 From: DerGamerPanda Date: Thu, 26 Jun 2025 00:47:22 +0200 Subject: [PATCH] Update README.md --- README.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8d27205..ede8079 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,27 @@ -# website-crawler -Multi-Site Keyword Crawler with Fulltext & Port Scanning +# 🕷️ Multi-Site Web Crawler with Fulltext & Port Scanning + +A powerful Python-based crawler that performs keyword discovery, fulltext extraction, and optional per-host port scanning — across multiple domains at once. + +## 🚀 Features + +- 🔍 **Multi-Site, Multi-Keyword Search** + Crawl multiple starting URLs and search for multiple terms in parallel. + +- 🧠 **Fulltext Capture** + Stores all visible page text for later NLP or forensic analysis. + +- 📊 **CSV Export** + All metadata — titles, URLs, IPs, keywords, fulltext, open ports — is exported to `.csv`. + +- 🔌 **Optional Custom Port Scan** + Scan user-defined ports on matched IP addresses to discover open services. + +- 🖥️ **GUI & CLI Interface** + - Use `crawler_gui.py` with a browser interface (Streamlit) + - Use `crawler_cli.py` for headless jobs or automation + +- 🐳 **Docker & Portainer Ready** + Ships with a `Dockerfile` and `docker-compose.yml` — deployable to cloud, local or container orchestration platforms. + + +