×
Login Register an account
Top Submissions Explore Upgoat Search Random Subverse Random Post Colorize! Site Rules Donate
5

A web crawler for a web search engine that indexes everything it can and doesn't listen to websites exclusion robots text which may mean you're going to get much more uncensored information

submitted by Crackinjokes to technology 1.7 yearsAug 10, 2023 12:33:52 ago (+5/-0)     (www.sogou.com)

https://www.sogou.com

Sogou.com

(Even though this is Chinese you can put English search words in and get websites back)

Is a search engine based on a web crawler which goes through all the websites on the web like most search engines except this one does not respect the robots.txt file that each website has which tells search engines what not to index.

Well the good thing about this is it means it's going to get deep into a lot of websites that other search engines won't get into..


Now I don't know whether they store your search information or whether they still censor results or anything like that and it's an Asian search engine and I don't know much more about it but I've already used it a few times in English even though it says you know it's writings are in Chinese and it produces the English websites

I've used it for a few things and it produces results that other websites don't.


I found it by doing a search for a list of web crawlers which are the robots that go out there and index all the websites and the article about the web crawlers mentioned that this one did not respect the robots.txt file which immediately caught my eye as a possible plus.


6 comments block


[ - ] beece 1 point 1.7 yearsAug 10, 2023 13:46:19 ago (+1/-0)

Thank you. I've noted that Yandex (russian) works better than google and bing for when I am searching fo the truth.

[ - ] Trope 0 points 1.7 yearsAug 10, 2023 16:10:33 ago (+0/-0)

I found Yandex to be a copypaste of all other search engines with minor differences.

Would love to see a primitive search engine that worked using literal tags and text strings. Archive.org is a fantastic example.

[ - ] allAheadFull 0 points 1.7 yearsAug 10, 2023 17:43:21 ago (+0/-0)

Depends on what is being censored. Compare the search "i hate jews" into Yandex vs. google, bing, etc.

Google won't even return results from sites like voat unless you use the "site:voat.xyz" directive. They didn't used to return results for voat even with it though.

[ - ] SecretHitler 1 point 1.7 yearsAug 10, 2023 12:56:45 ago (+1/-0)

Do you have an example of a website that hides its "censored" information behind robots.txt exclusions?

I don't think respecting robots.txt is the reason search sucks so bad.

[ - ] autotic 0 points 1.7 yearsAug 10, 2023 20:09:33 ago (+0/-0)

Not honoring robots.txt will get your indexer blocked.

[ - ] Terminalelektroshock 0 points 1.7 yearsAug 10, 2023 15:26:20 ago (+0/-0)

Uh oh better not tell AOU about this...she'll force users to introduce their introduction posts then algo them.