PagesXcrawler – Past Crawls

Last updated: 2026-03-05 19:53:19  ·  76 recorded crawl(s)

How to Use PagesXcrawler

PagesXcrawler is a GitHub-powered web crawler. Every crawl is triggered by a GitHub Issue or the Actions workflow dispatch UI, and results are published automatically to this GitHub Pages site.

Method 1 – GitHub Issue

  1. Go to the Issues tab and click New issue.
  2. Set the title using the format below and submit.
  3. The bot will comment with progress and close the issue when done.
crawl: https://example.com 3
crawl: https://example.com 2 --max-pages 50 --timeout 15

Method 2 – Workflow Dispatch

  1. Go to Actions → Web Crawler.
  2. Click Run workflow and fill in the fields.
  3. The results dashboard will update automatically after the run.
# fields available:
URL          – e.g. https://example.com
DEPTH        – e.g. 3
MAX_PAGES    – e.g. 100   (default)
TIMEOUT      – e.g. 10    (seconds, default)
ROTATE_AGENT – e.g. 10   (requests before agent rotation)

Command-Line / Issue Parameters

Parameter Required Default Description
url ✅ — The URL to crawl (must start with http:// or https://)
depth ✅ — Number of link levels to follow (1–5 recommended)
--max-pages ❌ 100 Maximum pages to crawl
--timeout ❌ 10 Per-request timeout in seconds
--rotate-agent-after ❌ 10 Requests between user-agent rotations

Past Crawls (76)

# Timestamp URL Depth Max Pages Pages Crawled Status
1 https://ddosecrets.com 166 ✓ completed
2 https://chat.deepseek.com 164 ✓ completed
3 https://www.2b2t.org 161 ✓ completed
4 https://www.2b2t.org 160 ✓ completed
5 https://github.com 159 ✓ completed
6 https://chat.deepseek.com 158 ✓ completed
7 https://chat.deepseek.com 157 ✓ completed
8 https://chat.deepseek.com 156 ✓ completed
9 https://ddosecrets.com 155 ✓ completed
10 https://ddosecrets.com 154 ✓ completed
11 https://github.com/unaveragetech 153 ✓ completed
12 https://example.com ✓ completed
13 https://example.com ✓ completed
14 https://ddosecrets.com:depth(2) 2 ✓ completed
15 https://valleywisehealth.org:depth(3) 3 ✓ completed
16 https://ddosecrets.com:depth(10) 10 ✓ completed
17 https://ddosecrets.com:depth(2) 2 ✓ completed
18 https://coolercrew.com/treasure_hunt/SyracuseWinterfest:depth(2) 2 ✓ completed
19 https://ddosecrets.com:depth(2) 2 ✓ completed
20 https://coolercrew.com/treasure_hunt/SyracuseWinterfest:depth(2) 2 ✓ completed
21 https://ddosecrets.com:depth(2) 2 ✓ completed
22 https://minewind.com/:depth(3) 3 ✓ completed
23 https://ddosecrets.com:depth(2) 2 ✓ completed
24 https://coolercrew.com/treasure_hunt/SyracuseWinterfest:depth(1) 1 ✓ completed
25 https://ddosecrets.com:depth(2) 2 ✓ completed
26 https://coolercrew.com/treasure_hunt/SyracuseWinterfest:depth(2) 2 ✓ completed
27 https://coolercrew.com/treasure_hunt/SyracuseWinterfest:depth(5) 5 ✓ completed
28 https://ddosecrets.com:depth(30) 30 ✓ completed
29 https://minewind.fandom.com/wiki/Essence:depth(6) 6 ✓ completed
30 https://minewind.com/:depth(3) 3 ✓ completed
31 https://minewind.com/:depth(5) 5 ✓ completed
32 https://minewind.com/:depth(5) 5 ✓ completed
33 https://4anime.gg/home:depth(3) 3 ✓ completed
34 https://4anime.gg/home:depth(3) 3 ✓ completed
35 https://4anime.gg/home:depth(10) 10 ✓ completed
36 https://www.google.com/:depth(3) 3 ✓ completed
37 https://sketch.io/sketchpad/:depth(5) 5 ✓ completed
38 https://optical.toys/:depth(4) 4 ✓ completed
39 https://puginarug.com:depth(3) 3 ✓ completed
40 https://github.com/unaveragetech/PagesXcrawler:depth(2) 2 ✓ completed
41 https://www.youtube.com:depth(10) 10 ✓ completed
42 https://www.youtube.com:depth(3) 3 ✓ completed
43 https://github.com/unaveragetech:depth(1) 1 ✓ completed
44 https://github.com/unaveragetech:depth(2) 2 ✓ completed
45 https://github.com/unaveragetech:depth(3) 3 ✓ completed
46 https://github.com/unaveragetech:depth(2) 2 ✓ completed
47 https://github.com/unaveragetech:depth(10) 10 ✓ completed
48 https://github.com/unaveragetech:depth(2) 2 ✓ completed
49 https://github.com/unaveragetech:depth(4) 4 ✓ completed
50 https://github.com/unaveragetech:depth(4) 4 ✓ completed
51 https://clicktheredbutton.com:depth(4) 4 ✓ completed
52 https://meteorclient.com:depth(10) 10 ✓ completed
53 https://www.netflix.com:depth(4) 4 ✓ completed
54 https://meteorclient.com:depth(10) 10 ✓ completed
55 https://www.netflix.com:depth(4) 4 ✓ completed
56 https://minewind.com:depth(5) 5 ✓ completed
57 https://www.youtube.com/watch?v=SX3x9pVSf-w:depth(10) 10 ✓ completed
58 https://puginarug.com:depth(3) 3 ✓ completed
59 https://www.youtube.com/watch?v=JqZrKtzZk7U:depth(2) 2 ✓ completed
60 https://www.google.com:3 3 ✓ completed
61 https://minewind.com:1 1 ✓ completed
62 https://optical.toys:4 4 ✓ completed
63 https://minewind.com:4 4 ✓ completed
64 https://minewind.com:2 2 ✓ completed
65 https://www.google.com:3 3 ✓ completed
66 https://minewind.com:10 10 ✓ completed
67 https://minewind.com:5 5 ✓ completed
68 https://www.youtube.com:3 3 ✓ completed
69 https://www.netflix.com:4 4 ✓ completed
70 https://www.netflix.com:2 2 ✓ completed
71 https://chatgpt.com:2 2 ✓ completed
72 https://www.youtube.com:2 2 ✓ completed
73 2026-03-05 18:54:28 https://ddosecrets.com 2 150 49 ✓ completed
74 2026-02-02 08:56:14 https://ddosecrets.com 2 150 49 ✓ completed
75 2026-02-02 08:41:26 https://ddosecrets.com 10 150 49 ✓ completed
76 2025-06-17 09:47:50 https://ddosecrets.com 3 50 51 ✓ completed

Rows are shown newest-first.