A high-performance web crawler and scanner — recursively crawls websites, respects robots.txt, handles rate-limiting, and produces comprehensive JSON or text reports. Available as a single, self-contained binary via npm.

DeepScanBot is a feature-rich, concurrent web crawler that makes website crawling fast, reliable, and developer-friendly. It handles the complexities of web crawling automatically:

• Discovers all pages on a website by following internal links up to a configurable depth
• Respects crawl rules defined in robots.txt with an option to bypass when needed
• Handles rate limits with automatic retry and exponential backoff
• Filters content by MIME type for targeted crawling
• Exports results in structured JSON or readable text formats with detailed analytics

• Security Researchers - Audit websites for exposed endpoints
• SEO Specialists - Discover all pages, find broken links, analyze site structure
• QA Engineers - Verify page availability, test website coverage
• Data Analysts - Collect structured data from websites for analysis
• DevOps Engineers - Monitor website health, validate deployments

• ⚡ Concurrent Crawling: Multi-threaded architecture with configurable concurrency, per-host rate limiting, and CPU-aware auto-scaling
• 🛡️ Robots.txt Compliance: Automatically respects robots.txt rules with optional bypass
• 🔄 Retry & Rate-Limit Handling: Automatic retry with exponential backoff and Retry-After header parsing
• 📊 Rich Output: JSON or text reports with detailed summaries, status code distribution, and skip reason breakdowns
• 🗺️ Sitemap Discovery: Auto-discover and crawl URLs from sitemap.xml, including nested sitemap indexes
• 📁 Content-Type Filtering: Filter downloads by MIME type, enforce page size limits
• 🌐 Proxy Support: Route traffic through HTTP/HTTPS proxy servers
• ▶️ Resume Mode: Resume interrupted crawls without recrawling already visited URLs
• 🌍 Cross-Domain Crawling: Optionally follow links to external domains
• 🔒 TLS Options: Disable TLS verification for self-signed certificates

npm install -g @mindfiredigital/deepscanbot

After installation, the deepscanbot command will be available globally:

deepscanbot --help

No additional runtime dependencies required. The npm package automatically installs the correct binary for your platform.

# Crawl a website with default settings
deepscanbot scan https://example.com

# Crawl with custom depth and concurrency
deepscanbot scan https://example.com depth=3 concurrency=10

# Output as JSON
deepscanbot scan https://example.com json=true

# Use a proxy
deepscanbot scan https://example.com proxy=http://127.0.0.1:8080

# Enable sitemap discovery
deepscanbot scan https://example.com sitemap=true

# Resume a previous crawl
deepscanbot scan https://example.com resume=true output=crawler_results

# Show version
deepscanbot version

# Verify installation
deepscanbot doctor

# Generate shell completion
deepscanbot completion bash

DeepScanBot supports a consistent --json flag across all commands that return structured data. When enabled, all output is written as valid JSON to stdout, while progress messages and diagnostics are sent to stderr.

DeepScanBot is designed to run reliably in CI/CD pipelines, scripts, and AI agents without ever hanging on user input.

Add --no-input to any command to disable all interactive prompts. If required input is missing, the CLI fails immediately with a clear error message instead of waiting.

# Run scan in non-interactive mode
deepscanbot scan https://example.com --no-input --force

# Check version non-interactively
deepscanbot --no-input version --json

# Run doctor in CI
deepscanbot --no-input doctor

The --force flag allows overwriting existing output files without prompting. In non-interactive mode, the scan command refuses to overwrite an existing output file unless --force is explicitly passed.

# Safe for CI/CD — overwrites output if it exists
deepscanbot scan https://example.com --no-input --force

# Will fail in non-interactive mode without --force
deepscanbot scan https://example.com --no-input
# Error: Output file "crawler_results.txt" already exists.
# Hint: Pass --force to overwrite or use output=<filename>.

When stdin is not connected to a terminal (e.g., piped input, CI runners), the CLI automatically detects the non-TTY environment and never waits for user input. The --no-input flag provides an explicit override for cases where TTY detection is insufficient.

# Always use --no-input and --force for automated runs
deepscanbot scan https://example.com --no-input --force --json

# Use exit codes to check success
if deepscanbot scan https://example.com --no-input --force depth=0; then
  echo "Scan completed"
else
  echo "Scan failed with exit code $?"
fi

DeepScanBot uses standardized exit codes to make CLI failures predictable for scripts, CI/CD pipelines, and AI agents. Every command returns a consistent exit code that tells you exactly what happened.

# Check exit code in a script
deepscanbot scan https://example.com
exit_code=$?
echo "Exit code: $exit_code"

# Conditional execution based on exit code
if deepscanbot scan https://example.com depth=0; then
  echo "Scan succeeded"
else
  echo "Scan failed with exit code $?"
fi

All errors include:

• What went wrong — a clear description of the problem
• Why it happened — the root cause when possible
• How to fix it — an actionable hint with an example

$ deepscanbot scan ftp://example.com
Error: Invalid URL: "ftp://example.com" must be an absolute http:// or https:// URL.
Hint: Example: https://example.com

$ deepscanbot scan http://example.com output=
Error: Output filename must not be empty.
Hint: Use output=<filename> with a non-empty value.

# Scan with JSON output
deepscanbot scan https://example.com --json

# Version with JSON output
deepscanbot version --json

# Doctor with JSON output
deepscanbot doctor --json

$ deepscanbot scan https://example.com depth=0 --json
{
  "status": "success",
  "data": {
    "start_url": "https://example.com",
    "output_file": "crawler_results.json",
    "started_at": "2024-01-15T10:30:00Z",
    "finished_at": "2024-01-15T10:30:05Z",
    "duration_ms": 5000,
    "summary": {
      "total": 1,
      "passed": 1,
      "failed": 0,
      "skipped": 0,
      "discovered": 0,
      "max_depth": 0
    },
    "urls": [
      {
        "url": "https://example.com",
        "depth": 0,
        "status_code": 200,
        "content_type": "text/html",
        "result": "passed"
      }
    ],
    "skipped": null
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:05Z",
    "command": "scan",
    "duration_ms": 5000
  }
}

$ deepscanbot version --json
{
  "status": "success",
  "data": {
    "version": "1.0.0",
    "name": "DeepScanBot CLI"
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:00Z",
    "command": "version",
    "duration_ms": 0
  }
}

$ deepscanbot doctor --json
{
  "status": "success",
  "data": {
    "installed": true,
    "executable": true,
    "configured": true,
    "checks_passed": 3,
    "message": "All checks passed!"
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:00Z",
    "command": "doctor",
    "duration_ms": 0
  }
}

$ deepscanbot scan invalid-url --json
{
  "status": "error",
  "error": {
    "message": "invalid URL \"invalid-url\": must be an absolute http:// or https:// URL",
    "code": "invalid_url"
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:00Z",
    "command": "scan",
    "duration_ms": 0
  }
}

• Valid JSON Only: stdout contains only valid JSON when --json is enabled
• Separate Streams: Progress messages, warnings, and logs are written to stderr
• Consistent Format: All commands use the same JSON response structure
• Backward Compatible: Existing behavior is preserved when --json is not specified
• Extensible: The centralized output formatting makes it easy to add new output formats in the future

All JSON responses follow this structure:

{
  "status": "success | error",
  "data": { ... },        // Present on success
  "error": {              // Present on error
    "message": "Error description",
    "code": "error_code"
  },
  "meta": {
    "timestamp": "ISO8601 timestamp",
    "command": "command_name",
    "duration_ms": 1234
  }
}

The JSON output mode is designed for easy integration with scripts and tools:

# Parse with jq
deepscanbot scan https://example.com --json | jq '.data.summary'

# Extract specific fields
deepscanbot version --json | jq -r '.data.version'

# Check command success
if deepscanbot scan https://example.com --json | jq -e '.status == "success"'; then
  echo "Scan completed successfully"
fi

DeepScanBot uses a modern command-based CLI structure similar to git, docker, and kubectl.

deepscanbot <command> [options]

deepscanbot scan <url> [key=value ...]

Options are specified as key=value pairs after the URL.

deepscanbot scan https://example.com

deepscanbot scan https://docs.example.com \
  depth=5 \
  concurrency=20 \
  host-concurrency=5 \
  timeout=10 \
  delay=200ms \
  retries=3 \
  retry-backoff=1s \
  json=true \
  sitemap=true \
  cross-domain=true \
  content-types="text/html application/pdf" \
  output=scan_results

The release process is fully automated via GitHub Actions.

Git Tag (v1.0.0)
      ↓
GitHub Action (release.yml)
      ↓
GoReleaser Build
      ↓
dist/ (binaries + checksums)
      ↓
Copy to npm package
      ↓
Update package.json version
      ↓
npm publish
      ↓
npm install -g @mindfiredigital/deepscanbot

1. Create a Git tag:

git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0

2. The GitHub Action will automatically:

   • Build binaries for all platforms via GoReleaser
   • Generate SHA256 checksums
   • Copy binaries into the npm package
   • Update package.json version from the Git tag
   • Publish to the npm registry
   • Create a GitHub Release with artifacts

3. Users can then install:

npm install -g @mindfiredigital/deepscanbot

This project follows Semantic Versioning:

• MAJOR (1.x.x): Breaking changes
• MINOR (x.1.x): New features, backward compatible
• PATCH (x.x.1): Bug fixes, backward compatible

Pre-release versions use the -next tag on npm:

npm install -g @mindfiredigital/deepscanbot@next

The npm publish configuration comes from environment variables:

# npmjs (default)
npm publish

# GitHub Packages
NPM_REGISTRY=https://npm.pkg.github.com/ npm publish

# Verdaccio (local)
NPM_REGISTRY=http://localhost:4873/ npm publish

# Dry run (verify without publishing)
npm publish --dry-run

For testing purposes, you can create snapshot releases:

# Create a snapshot build
goreleaser release --snapshot --clean

# Copy binaries and update version
bash scripts/copy-to-npm.sh
VERSION=0.0.0-snapshot node scripts/sync-version.js

# Dry run publish
npm publish --dry-run

project/
│
├── apps/
│   └── cli/              # Go CLI application
│       ├── main.go
│       └── tests/
│
├── packages/
│   ├── crawler/          # Web crawling logic
│   ├── exitcode/         # Standardized exit codes and error handling
│   ├── noinput/          # Non-interactive mode and TTY detection
│   ├── fetcher/          # HTTP fetching
│   ├── logger/           # Logging utilities
│   ├── output/           # Output formatting (JSON, human-readable, command tree)
│   ├── parser/           # HTML parsing
│   ├── storage/          # Output storage
│   └── types/            # Shared types
│
├── dist/                 # Pre-built binaries (generated)
├── bin/                  # Installed binary (generated by postinstall)
├── scripts/              # Helper scripts
│   └── prepublish-check.js
│
├── .github/
│   └── workflows/
│       ├── ci.yml             # CI workflow
│       └── release.yml        # Release workflow
│       └── release-docs.yml   # Release docs workflow
│
├── .goreleaser.yml       # GoReleaser configuration
├── package.json          # npm package configuration
├── postinstall.js        # npm post-install script
└── README.md             # This file

Problem: npm install -g @mindfiredigital/deepscanbot fails

Solutions:

• Ensure you have Node.js 18+ installed: node --version
• Check npm permissions: npm config get prefix
• Try with sudo (Unix): sudo npm install -g @mindfiredigital/deepscanbot
• Clear npm cache: npm cache clean --force

Problem: "Unsupported platform" error during installation

Solutions:

• Verify your OS and architecture: node -e "console.log(process.platform, process.arch)"
• Supported platforms: macOS (x64, arm64), Linux (x64, arm64), Windows (x64)
• Ensure you're using a 64-bit version of Node.js

Problem: "Command not found" after installation

Solutions:

• Check npm global bin directory is in your PATH: npm bin -g
• Add to PATH: export PATH=$(npm bin -g):$PATH
• On Windows, restart your terminal after installation

Problem: No results or empty output

Solutions:

• Verify the URL is accessible: curl -I https://example.com
• Increase timeout: deepscanbot scan <url> timeout=10
• Disable TLS verification for testing: deepscanbot scan <url> insecure=true
• Check if robots.txt is blocking: deepscanbot scan <url> ignore-robots=true

Problem: Too many requests or being blocked

Solutions:

• Reduce concurrency: deepscanbot scan <url> concurrency=2
• Add crawl delay: deepscanbot scan <url> delay=1s
• Use a proxy: deepscanbot scan <url> proxy=http://proxy.example.com:8080

Runs on every push and pull request to main:

• Builds Go code
• Runs tests with race detection
• Runs golangci-lint
• Checks Go formatting
• Verifies GoReleaser configuration
• Cross-platform build check
• npm package validation

Triggered by pushing a Git tag matching v*:

• Builds binaries via GoReleaser
• Copies binaries to npm package
• Updates package.json version
• Generates SHA256 checksums
• Verifies binaries
• Publishes to npm registry
• Uploads release artifacts

Please see CONTRIBUTING.md for contribution guidelines.

This project is licensed under the MIT License - see LICENSE.md for details.