Crawling wikipedia?


Is it possible to crawl wikipedia using the fess crawler? I have reduced the boost and interval time since wikipedia has some restrictions. But I haven’t been able to crawl their sites (only the main page is crawled and indexed)


Need more info… ex. what is your crawling configs?

Here is the configuration:

I tried putting on “Included URLs For Crawling”:* but it didn’t work either.

Also I created this job to schedule the crawling:

Interval time is too long.
I tried it and wikipedia pages were indexed.

What interval time are you using?

To check it in my environment, settings are:

Include URL:*
Interval time: 1000
Max Access Count: 10

Thanks a lot, everything appears to be working correctly.