>I am John Soyjak and this is my favorite forum of the soyshere.
  • Welcome. Registering and having your account be approved may take a while since there's only one guy checking and approving accounts and guestposts
  • Boards are being moved around and added. Leave feedback in this thread and let jannies move your threads

Shroomchiver - Shroom Thread Archiver

1789252638787.png
1789253492876.png
1789253708253.png

Used JewGPT (PBUH) to slowly build up the script since I awoke this morning. To list everything that has been added:
  • Queue handling now works in the one Chrome window.
  • Attachment downloading has been streamlined by the script making Chrome scroll up and down twice to make sure everything is loaded before sending out multiple workers to download it all at once as opposed to one at a time as it did before.
  • The script now boots Chrome with - to my knowledge - broadly compatible flags that disables any background processes and unnecessary graphical junk while still pleasing Cloudflare.
  • Exported HTML files now resemble Xenforo more.
  • Exported HTML files now have dates attached to posts.
  • Exported HTML files now have working search functionality that can filters by member(s), text content, and date(s)/ date-ranges in the entire thread by referring to a .js database for all posts, which is roughly 2.8MB per 500 posts (non-issue space-wise).
  • Some visual tweaks to the Shroomchiver interface itself.
  • More little loose tweaks to my slopcode as to fix some unoptimized shite from before.
All in all I'm pleased with it.

To-do's:
  • Toggle in the Shroomchiver to scrape signatures and user titles, that can then be toggled on and off individually in the HTML files themselves.
  • Better implementation of user awards.
  • Embed the proper "Soyjak.Forum" tiled background in the HTML files through data-uri base64 somehow. Maybe make a toggle for it.
  • Themes?
  • Try to fix the reaction bar to actually display the proper images as opposed to solely the text.
  • Fix emoticons from being large or not displaying at all due to how Xenforo handles them.
 
Last edited:
User1: why is everyone still awake

User2: because sleep is a government psyop

User3: i tried sleeping and then remembered that penguins exist

User1: what does that have to do with anything

User3: everything

User4: has anyone seen that video where the guy puts a traffic cone on a statue

User2: no but i need it immediately

User5: wait i have it

User5: [sends completely unrelated image]

User1: this is not the traffic cone

User5: close enough

User3: why does this actually look like something you'd see in a dream

User2: bro the statue looks disappointed

User4: he knows what you did

User1: i didn't do anything

User4: that's exactly what someone who did something would say

User6: anyway does anyone know if CachyOS is actually good

User2: we're discussing statues

User6: yes but answer the question

User3: linux users when someone asks them a normal question: 47 paragraphs about package managers

User6: i hate all of you

User1: update: i found the traffic cone video

User2: SEND

User1: it's literally just a traffic cone on a statue

User3: cinema

User5: absolute cinema

User4: humanity peaked here

User6: can we go back to talking about linux

User2: no
 
kriper2004 said:
User1: why is everyone still awake

User2: because sleep is a government psyop

User3: i tried sleeping and then remembered that penguins exist

User1: what does that have to do with anything

User3: everything

User4: has anyone seen that video where the guy puts a traffic cone on a statue

User2: no but i need it immediately

User5: wait i have it

User5: [sends completely unrelated image]

User1: this is not the traffic cone

User5: close enough

User3: why does this actually look like something you'd see in a dream

User2: bro the statue looks disappointed

User4: he knows what you did

User1: i didn't do anything

User4: that's exactly what someone who did something would say

User6: anyway does anyone know if CachyOS is actually good

User2: we're discussing statues

User6: yes but answer the question

User3: linux users when someone asks them a normal question: 47 paragraphs about package managers

User6: i hate all of you

User1: update: i found the traffic cone video

User2: SEND

User1: it's literally just a traffic cone on a statue

User3: cinema

User5: absolute cinema

User4: humanity peaked here

User6: can we go back to talking about linux

User2: no
Click to expand...
average conversation according to chatlgbt, after feeding him an archived thread
 
cargo said:
Forgot to post this. Toggleable themes, user banners (Moderator, TOP POSTER OF THE WEEK, etc), signatures and usertitles.
View attachment 171156
Click to expand...
I can confirm GNU/Linux support does work with both Chrome and Chromium, albeit with the following fix:
Python: 
def make_driver(PROFILE_DIR):
    PROFILE_DIR = os.path.abspath(PROFILE_DIR.replace("\\", "/"))
    os.makedirs(PROFILE_DIR, exist_ok=True)

    options_uc = uc.ChromeOptions()
    options_uc.add_argument("--disable-gpu")
    options_uc.add_argument("--disable-smooth-scrolling")
    options_uc.add_argument("--disable-backgrounding-occluded-windows")
    options_uc.add_argument("--disable-background-timer-throttling")
    options_uc.add_argument("--disable-background-networking")
    options_uc.add_argument("--disable-component-update")
    options_uc.add_argument("--disable-domain-reliability")
    options_uc.add_argument("--disable-sync")
    options_uc.add_argument("--disable-preconnect")
    options_uc.add_argument("--dns-prefetch-disable")
    options_uc.add_argument("--no-first-run")
    options_uc.add_argument("--no-default-browser-check")
    options_uc.add_argument("--autoplay-policy=user-gesture-required")
    options_uc.add_argument("--no-sandbox")
    options_uc.add_argument("--disable-dev-shm-usage")

    prefs = {
        "profile.default_content_setting_values.notifications": 2,
        "profile.default_content_setting_values.geolocation": 2,
        "profile.default_content_setting_values.media_stream": 2,
    }
    options_uc.add_experimental_option("prefs", prefs)

    if sys.platform.startswith("linux"):
        for candidate in (
            "/usr/bin/google-chrome-stable",
            "/usr/bin/google-chrome",
            "/usr/bin/chromium-browser",
            "/usr/bin/chromium",
        ):
            if os.path.isfile(candidate) and os.access(candidate, os.X_OK):
                options_uc.binary_location = candidate
                print(f"Using Chrome binary: {candidate}")
                break
        else:
            print("WARNING: no Chrome/Chromium binary found on PATH/common locations")

    major = get_chrome_major_version()
    print(f"Detected Chrome major version: {major}")

    last_exc = None
    try:
        kwargs = {
            "options": options_uc,
            "headless": False,
            "use_subprocess": True,
            "user_data_dir": PROFILE_DIR,
        }
        if major is not None and major < 200:
            kwargs["version_main"] = major
        return uc.Chrome(**kwargs)
    except Exception as e:
        last_exc = e
        m = re.search(r"only supports Chrome version\s+(\d+)", str(e), re.I)
        if m:
            try:
                kwargs["version_main"] = int(m.group(1))
                return uc.Chrome(**kwargs)
            except Exception as e2:
                last_exc = e2
    raise last_exc
 
cargo said:
I can confirm GNU/Linux support does work with both Chrome and Chromium, albeit with the following fix:
Python: 
def make_driver(PROFILE_DIR):
    PROFILE_DIR = os.path.abspath(PROFILE_DIR.replace("\\", "/"))
    os.makedirs(PROFILE_DIR, exist_ok=True)

    options_uc = uc.ChromeOptions()
    options_uc.add_argument("--disable-gpu")
    options_uc.add_argument("--disable-smooth-scrolling")
    options_uc.add_argument("--disable-backgrounding-occluded-windows")
    options_uc.add_argument("--disable-background-timer-throttling")
    options_uc.add_argument("--disable-background-networking")
    options_uc.add_argument("--disable-component-update")
    options_uc.add_argument("--disable-domain-reliability")
    options_uc.add_argument("--disable-sync")
    options_uc.add_argument("--disable-preconnect")
    options_uc.add_argument("--dns-prefetch-disable")
    options_uc.add_argument("--no-first-run")
    options_uc.add_argument("--no-default-browser-check")
    options_uc.add_argument("--autoplay-policy=user-gesture-required")
    options_uc.add_argument("--no-sandbox")
    options_uc.add_argument("--disable-dev-shm-usage")

    prefs = {
        "profile.default_content_setting_values.notifications": 2,
        "profile.default_content_setting_values.geolocation": 2,
        "profile.default_content_setting_values.media_stream": 2,
    }
    options_uc.add_experimental_option("prefs", prefs)

    if sys.platform.startswith("linux"):
        for candidate in (
            "/usr/bin/google-chrome-stable",
            "/usr/bin/google-chrome",
            "/usr/bin/chromium-browser",
            "/usr/bin/chromium",
        ):
            if os.path.isfile(candidate) and os.access(candidate, os.X_OK):
                options_uc.binary_location = candidate
                print(f"Using Chrome binary: {candidate}")
                break
        else:
            print("WARNING: no Chrome/Chromium binary found on PATH/common locations")

    major = get_chrome_major_version()
    print(f"Detected Chrome major version: {major}")

    last_exc = None
    try:
        kwargs = {
            "options": options_uc,
            "headless": False,
            "use_subprocess": True,
            "user_data_dir": PROFILE_DIR,
        }
        if major is not None and major < 200:
            kwargs["version_main"] = major
        return uc.Chrome(**kwargs)
    except Exception as e:
        last_exc = e
        m = re.search(r"only supports Chrome version\s+(\d+)", str(e), re.I)
        if m:
            try:
                kwargs["version_main"] = int(m.group(1))
                return uc.Chrome(**kwargs)
            except Exception as e2:
                last_exc = e2
    raise last_exc
Click to expand...
Yeesh. Current codebase can hang Chromium due to how fast it goes and for whatever reason it always assumes the working directory is /home/"USERNAME" regardless of where it is ran. I'll see to these at some stage.
 
cargo said:
Yeesh. Current codebase can hang Chromium due to how fast it goes and for whatever reason it always assumes the working directory is /home/"USERNAME" regardless of where it is ran. I'll see to these at some stage.
Click to expand...
Code: 
========================================================================
THREAD 23/1151
https://soyjak.forum/threads/are-mres-good-for-malnutrition.4676
========================================================================
No existing pages found. Starting from page 1.
Loading thread start URL: https://soyjak.forum/threads/are-mres-good-for-malnutrition.4676

✗ Failed thread 23/1151: ReadTimeoutError: HTTPConnectionPool(host='localhost', port=37875): Read timed out. (read timeout=120)
Traceback (most recent call last):
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/lib64/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.9/socket.py", line 716, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/run/media/wobbles/SHROOM/_Shroomchiver2.py", line 1937, in _archive_worker
    print(target_url)
  File "/run/media/wobbles/SHROOM/_Shroomchiver2.py", line 1385, in archive_thread
    f"Browser tab verification failed for {TARGET_URL}"
  File "/home/wobbles/.local/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 665, in get
    return super().get(url)
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 483, in get
    self.execute(Command.GET, {"url": url})
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 455, in execute
    response = cast(RemoteConnection, self.command_executor).execute(driver_command, params)
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/remote_connection.py", line 405, in execute
    return self._request(command_info[0], url, body=data)
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/remote_connection.py", line 429, in _request
    response = self._conn.request(method, url, body=body, headers=headers, timeout=self._client_config.timeout)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/_request_methods.py", line 143, in request
    return self.request_encode_body(
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/_request_methods.py", line 278, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/poolmanager.py", line 457, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/util/retry.py", line 490, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=37875): Read timed out. (read timeout=120)
[chudthink]
 
cargo said:
Code: 
========================================================================
THREAD 23/1151
https://soyjak.forum/threads/are-mres-good-for-malnutrition.4676
========================================================================
No existing pages found. Starting from page 1.
Loading thread start URL: https://soyjak.forum/threads/are-mres-good-for-malnutrition.4676

✗ Failed thread 23/1151: ReadTimeoutError: HTTPConnectionPool(host='localhost', port=37875): Read timed out. (read timeout=120)
Traceback (most recent call last):
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/lib64/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.9/socket.py", line 716, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/run/media/wobbles/SHROOM/_Shroomchiver2.py", line 1937, in _archive_worker
    print(target_url)
  File "/run/media/wobbles/SHROOM/_Shroomchiver2.py", line 1385, in archive_thread
    f"Browser tab verification failed for {TARGET_URL}"
  File "/home/wobbles/.local/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 665, in get
    return super().get(url)
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 483, in get
    self.execute(Command.GET, {"url": url})
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 455, in execute
    response = cast(RemoteConnection, self.command_executor).execute(driver_command, params)
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/remote_connection.py", line 405, in execute
    return self._request(command_info[0], url, body=data)
  File "/home/wobbles/.local/lib/python3.9/site-packages/selenium/webdriver/remote/remote_connection.py", line 429, in _request
    response = self._conn.request(method, url, body=body, headers=headers, timeout=self._client_config.timeout)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/_request_methods.py", line 143, in request
    return self.request_encode_body(
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/_request_methods.py", line 278, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/poolmanager.py", line 457, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/util/retry.py", line 490, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/wobbles/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=37875): Read timed out. (read timeout=120)
[chudthink]
Click to expand...
Peculiar. I can't recreate it.
 
Scraping speeds don't fluctuate much with hardware from the past twenty years from what I can observe. My T430 runs a mere second faster than my T400 when scraping and compiling threads despite the modest jump in power. Also to note, I should implement working poll scraping and "Answer" scraping for threads that feature them.
 
Last edited:

Similar threads

Meursault(1)
Pepe and buster visit the shroom soyjak story
    • Pickle Rick
    • Jartycuck
Replies
0
Views
25
Meursault(1)
Meursault(1)
grimmki
  • Suggestion Suggestion
make the best jewtube ad for the shroom
    • Groovy!
    • No Words...
2 3 4
Replies
79
Views
464
FartmingerFuckwadCheesest
FartmingerFuckwadCheesest
absolutely not dino
  • Featured
Shroom OC general
    • Giga
8 9 10
Replies
197
Views
3K
randal
randal
OrHoweverTheBurgerIsEaten
there are almost 500 guests on the shroom right now
    • Breathtaking
Replies
4
Views
62
Ifuggenhatesaltmaynee
Lateral
  • Poll Poll
Shroom Users Tier List 2: Electric Boogaloo
    • Like
    • Gem
17 18 19
Replies
361
Views
6K
cirno.enthusiast 2
cirno.enthusiast 2

Members online

... and 3 more.
Total: 524 (members: 53, guests: 471)

Latest posts

Back
Top Bottom