Crawling web pages without encoding information.

English:Fess
1

(from github.com/biligee)
How do I crawl web pages that didn’t explicitly specify the encoding and text is encoded in shift-jis

2

(from github.com/marevol)
The default encoding is UTF-8.

3

(from github.com/biligee)
Yes. UTF-8 websites can be crawled. I guess, there are tons of websites that have <META content="text/html; charset=shift_jis" http-equiv=Content-Type>. How do I crawl them.