Question with writing a script

[Onzikin]

I'm pretty tech illeterate, but I wanna make a script that compares page A and page B and checks for which words are repeated between them. I will ask ai to vibecode it for me, but I wanna ask if there's anything else I need to know.

 

[Onzikin]

Jannies move this to /tech/ since that dnb board needs more threads.

 

[YellowLee]

Jannies move this to /tech/ since that dnb board needs more threads.

DM any online mod and they'll do it (with your reasoning of course)

 

[Apeson]

I'm pretty tech illeterate, but I wanna make a script that compares page A and page B and checks for which words are repeated between them. I will ask ai to vibecode it for me, but I wanna ask if there's anything else I need to know.

Depends on what you're checking. If it's for a wiki it's easier since you can just use the source. If it's for any other website you can just compare htmls though you will have a hard time with live sites that way.

 

[Apeson]

Depends on what you're checking. If it's for a wiki it's easier since you can just use the source. If it's for any other website you can just compare htmls though you will have a hard time with live sites that way.

press crtl + a to see what i mean.

 

[Apeson]

press crtl + a to see what i mean.

from there, you can filter by only the <meta name="description" content="example text here noob" /> tag, using the example of this site

 

[cirno.enthusiast 2]

Depends on what you're checking. If it's for a wiki it's easier since you can just use the source. If it's for any other website you can just compare htmls though you will have a hard time with live sites that way.

Kind of a mix of the two actually, kek. I wanted to compare a wiki page with text from wayback machine pages.

 

[Apeson]

Kind of a mix of the two actually, kek. I wanted to compare a wiki page with text from wayback machine pages.

then just use the html and look for a limiting tag that you can use the filter all the irrelevant bullshit out

 

[cirno.enthusiast 2]

A practical Python script can fetch multiple pages, extract their main text, and find repeated phrases (e.g., via n-grams) that appear across them.Here’s a clean, self-contained approach using only common libraries (requests, beautifulsoup4, and the standard library). It looks for exact repeated phrases of a chosen length that appear on at least N of the pages

pip install requests beautifulsoup4
import requests<br>from bs4 import BeautifulSoup<br>from collections import defaultdict<br>from urllib.parse import urlparse<br>import re<br>from typing import List, Dict, Set<br><br>def fetch_clean_text(url: str, timeout: int = 15) -&gt; str:<br> """Fetch a page and return cleaned visible text."""<br> headers = {<br> "User-Agent": "Mozilla/5.0 (compatible; TextComparator/1.0)"<br> }<br> try:<br> resp = requests.get(url, headers=headers, timeout=timeout)<br> resp.raise_for_status()<br> soup = BeautifulSoup(resp.content, "html.parser")<br><br> # Remove non-content elements<br> for tag in soup(["script", "style", "noscript", "header", "footer",<br> "nav", "aside", "form", "iframe"]):<br> tag.decompose()<br><br> text = soup.get_text(separator=" ", strip=True)<br> # Normalize whitespace and lowercase for matching<br> text = re.sub(r"\s+", " ", text).strip().lower()<br> return text<br> except Exception as e:<br> print(f"Failed to fetch {url}: {e}")<br> return ""<br><br>def get_ngrams(text: str, n: int) -&gt; Set[str]:<br> """Return set of word n-grams from text."""<br> words = text.split()<br> if len(words) &lt; n:<br> return set()<br> return {" ".join(words[i:i+n]) for i in range(len(words) - n + 1)}<br><br>def find_repeated_phrases(<br> urls: List[str],<br> ngram_size: int = 5, # phrase length in words<br> min_pages: int = 2, # must appear on at least this many pages<br> min_phrase_length: int = 20 # ignore very short matches (characters)<br>) -&gt; Dict[str, List[str]]:<br> """<br> Returns a dict: phrase → list of URLs where it appears.<br> """<br> page_texts = {}<br> for url in urls:<br> text = fetch_clean_text(url)<br> if text:<br> page_texts = text<br> print(f"Fetc...se I was too lazy to get calude just for this

 

[Barry]

I'm pretty tech illeterate, but I wanna make a script that compares page A and page B

Kid named diff

and checks for which words are repeated between them. I will ask ai to vibecode it for me, but I wanna ask if there's anything else I need to know.

Idk but I know linux users can still use a simple bash script for this