java.net.NoRouteToHostException

English:Fess
1

OK - I know what it means, but I am trying to understand & troubleshoot why it is reporting so.

Web crawls to externally hosted sites, system.abc.com & system.def.com run successfully from fess.xyz.com. However, crawls from fess.xyz.com to system.xyz.com reports java.net.NoRouteToHostException and the error message in fess-crawler

DEBUG The url is null.

and of course the 0 docs message, as to be expected.

From fess.xyz.com, I can ping system.xyz.com, I can do a curl https://system.xyz.com, but a webcrawl via fess does not work. I run wireshark & tcpdumps and I do see port 53 dns traffic, but even then it fails. I add the ip addr to /etc/hosts to eliminate dns lookup, same fail.

Now, whilst fess succeeds in crawling system.def.com, system.def.com is actually a hosted site on system.xyz.com.

If I can understand how & why fess.xyz cannot see/connect to system.xyz.com, then all the web crawls that were working 2 weeks ago should be OK, as there are all hosted on system.xyz.com

Thanks for your time.

Fin

2

What is the stack trace?

3

Hi Shinsuke, thanks…

debug or all level logging in fess-crawler.log? I can rerun a fresh test.

Fin

4

The current information is not enough to solve it. So, the stack trace, log messages, or the like are helpful.

5

image
image1013×868 13.4 KB

image
image949×292 7.44 KB

image
image791×229 4.03 KB

fess-crawler.log for cefims (successful crawl)

2021-04-11 17:14:39,795 [WebFsCrawler] INFO Target URL: https://www.cefims.ac.uk/
2021-04-11 17:14:39,795 [WebFsCrawler] INFO Included URL: https://www.cefims.ac.uk/.*
2021-04-11 17:14:39,795 [WebFsCrawler] INFO Excluded URL: https://www.cefims.ac.uk/cgi-bin/.*
2021-04-11 17:14:39,795 [WebFsCrawler] INFO Excluded URL: (?i).(php|mp3|gif|jpg|jpeg|JPEG|JPG|png|css|js|xml|rss|json|ppt|pptx|pps|ppsx|doc|docx|rtf|xls|xlsx)
2021-04-11 17:14:39,795 [WebFsCrawler] INFO Excluded URL: .-rss-.*
2021-04-11 17:14:39,803 [WebFsCrawler] DEBUG Crawling https://www.cefims.ac.uk/
2021-04-11 17:14:39,819 [IndexUpdater] DEBUG Starting indexUpdater.
2021-04-11 17:14:39,877 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Queued URL: [UrlQueueImpl [id=AVyq_lUmHWyes_9AjA98-1.aHR0cHM6Ly93d3cuY2VmaW1zLmFjLnVrLw, sessionId=AVyq_lUmHWyes_9AjA98-1, method=GET, url=https://www.cefims.ac.uk/, encoding=null, parentUrl=null, depth=0, lastModified=0, createTime=1618157679764]]
2021-04-11 17:14:39,885 [Crawler-AVyq_lUmHWyes_9AjA98-1-4] DEBUG The url is null. (0)
2021-04-11 17:14:39,886 [Crawler-AVyq_lUmHWyes_9AjA98-1-1] DEBUG The url is null. (0)
2021-04-11 17:14:39,890 [Crawler-AVyq_lUmHWyes_9AjA98-1-2] DEBUG The url is null. (0)
2021-04-11 17:14:39,901 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] INFO Crawling URL: https://www.cefims.ac.uk/
2021-04-11 17:14:39,903 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Searching indexed document: https:%2F%2Fwww.cefims.ac.uk%2F;role=Rguest
2021-04-11 17:14:39,905 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Query DSL:
2021-04-11 17:14:39,912 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Getting the content from URL: https://www.cefims.ac.uk/
2021-04-11 17:14:39,918 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Initializing org.codelibs.fess.crawler.client.http.HcHttpClient
2021-04-11 17:14:39,958 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Accessing https://www.cefims.ac.uk/
2021-04-11 17:14:39,958 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] INFO Checking URL: https://www.cefims.ac.uk/robots.txt
2021-04-11 17:14:39,967 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG CookieSpec selected: default
2021-04-11 17:14:39,980 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection request: [route: {s}->https://www.cefims.ac.uk:443][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 200]
2021-04-11 17:14:39,996 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection leased: [id: 0][route: {s}->https://www.cefims.ac.uk:443][total kept alive: 0; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 17:14:39,998 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Opening connection {s}->https://www.cefims.ac.uk:443
2021-04-11 17:14:40,006 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connecting to www.cefims.ac.uk/212.219.139.54:443
2021-04-11 17:14:40,006 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connecting socket to www.cefims.ac.uk/212.219.139.54:443 with timeout 0
2021-04-11 17:14:40,043 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
2021-04-11 17:14:40,043 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2021-04-11 17:14:40,043 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Starting handshake
2021-04-11 17:14:40,141 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Secure session established
2021-04-11 17:14:40,141 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG negotiated protocol: TLSv1.2
2021-04-11 17:14:40,141 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2021-04-11 17:14:40,141 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG peer principal: CN=www.cefims.ac.uk, O=School of Oriental and African Studies, L=London, ST=“London, City of”, C=GB, SERIALNUMBER=1916-05-23, OID.2.5.4.15=Government Entity, OID.1.3.6.1.4.1.311.60.2.1.3=GB
2021-04-11 17:14:40,141 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG peer alternative names: [www.cefims.ac.uk, cefims.ac.uk, dev.cefims.ac.uk]
2021-04-11 17:14:40,142 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG issuer principal: CN=QuoVadis Europe EV SSL CA G1, O=QuoVadis Trustlink B.V., C=NL
2021-04-11 17:14:40,145 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection established 212.219.238.219:37918<->212.219.139.54:443
2021-04-11 17:14:40,145 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Executing request GET /robots.txt HTTP/1.1
2021-04-11 17:14:40,145 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Target auth state: UNCHALLENGED
2021-04-11 17:14:40,146 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Proxy auth state: UNCHALLENGED
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> GET /robots.txt HTTP/1.1
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> Host: www.cefims.ac.uk
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> Connection: Keep-Alive
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> User-Agent: Mozilla/5.0 (compatible; Fess/12.0; +http://fess.codelibs.org/bot.html)
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> Accept-Encoding: gzip,deflate
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “GET /robots.txt HTTP/1.1[\r][\n]”
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “Host: www.cefims.ac.uk[\r][\n]”
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]”
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “User-Agent: Mozilla/5.0 (compatible; Fess/12.0; +http://fess.codelibs.org/bot.html)[\r][\n]”
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “Accept-Encoding: gzip,deflate[\r][\n]”
2021-04-11 17:14:40,147 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “HTTP/1.1 200 OK[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Server: nginx[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Date: Sun, 11 Apr 2021 16:14:40 GMT[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Type: text/plain[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Length: 155[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Connection: keep-alive[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Last-Modified: Thu, 15 Jun 2017 11:41:31 GMT[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “ETag: “5942726b-9b”[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-Proxy-Cache: REVALIDATED[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Security-Policy: default-src https: ‘unsafe-eval’ ‘unsafe-inline’; object-src ‘none’[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Referrer-Policy: origin[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-Content-Type-Options: nosniff[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Security-Policy: frame-ancestors ‘none’[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-Frame-Options: DENY[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-XSS-Protection: 1; mode=block[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Accept-Ranges: bytes[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “User-agent: [\n]"
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Disallow: /cgi-bin/events_edit.cgi[\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Disallow: /cgi-bin/programmes_edit.cgi[\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Disallow: /cgi-bin/research_edit.cgi[\n]”
2021-04-11 17:14:40,154 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Disallow: /cgi-bin/search.cgi[\n]”
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << HTTP/1.1 200 OK
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Server: nginx
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Date: Sun, 11 Apr 2021 16:14:40 GMT
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Content-Type: text/plain
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Content-Length: 155
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Connection: keep-alive
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Last-Modified: Thu, 15 Jun 2017 11:41:31 GMT
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << ETag: “5942726b-9b”
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << X-Proxy-Cache: REVALIDATED
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Content-Security-Policy: default-src https: ‘unsafe-eval’ ‘unsafe-inline’; object-src ‘none’
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Referrer-Policy: origin
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << X-Content-Type-Options: nosniff
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Content-Security-Policy: frame-ancestors ‘none’
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << X-Frame-Options: DENY
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << X-XSS-Protection: 1; mode=block
2021-04-11 17:14:40,157 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << Accept-Ranges: bytes
2021-04-11 17:14:40,162 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection can be kept alive indefinitely
2021-04-11 17:14:40,167 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection [id: 0][route: {s}->https://www.cefims.ac.uk:443] can be kept alive indefinitely
2021-04-11 17:14:40,167 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0: set socket timeout to 0
2021-04-11 17:14:40,167 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection released: [id: 0][route: {s}->https://www.cefims.ac.uk:443][total kept alive: 1; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 17:14:40,231 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG CookieSpec selected: default
2021-04-11 17:14:40,231 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection request: [route: {s}->https://www.cefims.ac.uk:443][total kept alive: 1; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 17:14:40,231 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Connection leased: [id: 0][route: {s}->https://www.cefims.ac.uk:443][total kept alive: 0; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 17:14:40,231 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0: set socket timeout to 0
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Executing request GET / HTTP/1.1
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Target auth state: UNCHALLENGED
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG Proxy auth state: UNCHALLENGED
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> GET / HTTP/1.1
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> Host: www.cefims.ac.uk
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> Connection: Keep-Alive
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> User-Agent: Mozilla/5.0 (compatible; Fess/12.0; +http://fess.codelibs.org/bot.html)
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> Accept-Encoding: gzip,deflate
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “GET / HTTP/1.1[\r][\n]”
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “Host: www.cefims.ac.uk[\r][\n]”
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]”
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “User-Agent: Mozilla/5.0 (compatible; Fess/12.0; +http://fess.codelibs.org/bot.html)[\r][\n]”
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “Accept-Encoding: gzip,deflate[\r][\n]”
2021-04-11 17:14:40,232 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 >> “[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “HTTP/1.1 200 OK[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Server: nginx[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Date: Sun, 11 Apr 2021 16:14:40 GMT[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Type: text/html[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Transfer-Encoding: chunked[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Connection: keep-alive[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Vary: Accept-Encoding[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Expires: Sun, 11 Apr 2021 16:29:40 GMT[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Cache-Control: max-age=900[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Cache-Control: public[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Encoding: gzip[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-Proxy-Cache: EXPIRED[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Security-Policy: default-src https: ‘unsafe-eval’ ‘unsafe-inline’; object-src ‘none’[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Referrer-Policy: origin[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-Content-Type-Options: nosniff[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “Content-Security-Policy: frame-ancestors ‘none’[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-Frame-Options: DENY[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “X-XSS-Protection: 1; mode=block[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 17:14:40,243 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “198d[\r][\n]”
2021-04-11 17:14:40,244 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << “[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x3][0xdd]<is[0xdb]8[0xb2][0x9f][0xe3][0x81]h[0xab][0xc6][0xf6]>[0x1d][0xbe][0xe2]d"KS[0x8e][0xed]8[0x9e][0xc9]9vv&[0x9b]J[0xa5] [0x12][0x12]a[0x91][0x0]C[0x80][0x92][0x95][0x99][0xf9][0xef][0xaf][0x1b][0x0][0xf][0xd1][0x94],%[0xde]y[0xaf]6U[0xb1]H[0x10]h4[0xd0][0x7][0xfa][0x0]p[0xf4][0xf0][0xf4][0xcd][0xc9][0xd5][0x87][0xb7]g$[0xd0]Q[0xd8][0xdf]8[0xc2][0x1f][0x12]R1[0xea]5[0x98]h[0x1][0xa3]>[0xfc]DLS[0xe2][0x5]4QL[0xf7][0x1a][0xef][0xaf][0x9e][0xb7][0x9e][0xe0]W[0xcd]u[0xc8][0xfa]'L[0xe8][0x84][0x91][0xa1]L[0xc8]s.[0xa8][0xf0]8[\r][0x9][0x15]>yE[0x5][0x1d][0xb1][0x8]>[0x93]K[0x9d][0xfa][0x9c])[0xf2]'[0xb9]|s|I[0xde][0xb]>a[0x89][0xe2]zF[0xe4][0x90][0xbc][0x94][0xc2][0x97][0xe2][0xa8]c[0xc1]ml[0x1c]=l[0xb5]^[0x9c][0x1d][0x9f][0x9e][0xfd][0xda]je[0x9d][0xb][0x1a][0xb1]^c[0xcc]fS[0x99][0xf8][0xaa]A<)4[0x0][0xee]5N[0xd8]s[0xfe][0xea][0xb2]I[0xd6][0xc0][0xa2]IN[0xb9][0xd2]P[0x85][0x91][0x97][0x8c]&[0x82][0x8b]Q[0x93][0xbc][0x95]J[0x8f][0x12][0xea][0xa7]T[0xb3]&yu[0xe9]5[0xc9][0xd0][0xc0]a[0xd9][0x3][0x2][0xc][0xe9][0xb4]I[0xe2]t[0x10]r[0x8f][0xc4][0x12][0xfe][0xce][0x9a]$[0xca];h.[0x1e]]c~ [0x13][0xce][0xa6][0xb1]Lti S[0xee][0xeb][0xa0][0xe7][0xb3][0x9][0xf7]X[0xcb][0xbc]4[0xb9][0xe0][0x1a]zm)[0x8f][0x86][0xac][0xb7][0xdb][0xde][0xc9][0xa1][0x4]Z[0xc7]-[0xf6]%[0xe5][0x93]^[0xe3][0xf7][0xd6][0xfb][0xe3][0xd6][0x89][0x8c]b[0xaa][0xf9] d%[0x90][0x17]g=[0xe6][0x8f]X[0xc3][0xcd][0xe9][0xa5][0x9e][0x85]L[0x99]9[\r][0xb9][0x18][0x93][0x84][0x85][0xbd][0x86]2[0x85][0x1]c[0x80]K[0x90][0xb0]a[0xaf][0xd1]1E[0xf6][0xaf]j{[\n]" 2021-04-11 17:14:40,244 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << "f[[0xcf]b@Z[0xb3][0x1b][0xdd]1[0xef][0x11][0xf3]9[0x85][0xb6]^[0xc2],[0x9f][0xc0][0x13][0x8f]5Q[0x89][0xd7]k n[0xea]i[0xa7][0xe3][0xf9][0xe2][0x1a][0xda][0x87]2[0xf5][0x87]!MX[0xdb][0x93]Q[0x87]^[0xd3][0x9b]N[0xc8][0x7][0xaa][0x13]I[0x9f][0xc1][0xdc]M:{[0xed][0xc3][0xf6]^[0xf1][0xde][0x8e][0xb8]h_C/[0x1c][0x86]1J[0x1a][0xa1][0xa7][0x80][0xee]?9hM[0xce][0xde][0xa9][0xc3][0xc9]/[0x7]O&[0x87][0xff]s[0x1e][0x9f][0x9f][0x9e][0x88][0xdf][0xdf][0xbf];[0xa4]t[0xef][0xf4][0x83][0x16][0x8f]^^[0xf1][0x8e]>[0xee]<y[0xc4][0xcf][0xfe]=[0xfc][0xfd][0xf5][0x8e]8[0xfe]p[0xbd][0xa3][0x82][0xc9][0x7][0xf9]$[0xf4][0xdf][0xbd]{[0x1c][0xc1][0xdc]$R)[0x99][0xf0][0x11][0x17][0xbd][0x6][0x15]R[0xcc]”[0x99][0xaa]F[0xff][0xa8]c[0x87]P[0x19]K[0xe7][0x1a][0x11]E[0xda][0xba][0x1f]D[0xac]T[0xd9][0xf2][0xea][0xd5][0xab][0x97][0x8f][0xc8][0xc5][0x19]0[0xde][0x8d][0x99]]([0xfb][0xc8][0x87]$[0xd4]X[0xf8][0xe3][0xa7]~[0xed][0xf4][0xa0][0xac]=R[0x1][0x8f][0xda]#)G![0xf3][0xfc]f[0x8a][0xd4]Dtt[0x92][0x8a][0xb1][0xad]2[0xdf][0xe3][0xd1][0xc3][0x8f]L[0xf8]|[0xf8][0x9];2==[0x7]j[0xdf]M[0xd4][0xac][0xd7]T[0xb1][0xf6][0x10]Z[0xd0])S0 [0xd3][0xe1][0x81][0xc7][0x9e][0xec]<b[0xfb][0x87]CC[0xed]|[\n]" 2021-04-11 17:14:40,244 [Crawler-AVyq_lUmHWyes_9AjA98-1-3] DEBUG http-outgoing-0 << "[0xfa][0x1b]P_@[\r]#<(K[0xa0][0x2][0xc4][0x88][0x11][0x1d]pE[0xb4]$C[0x1a]*F[0xa4]3[0x99]n[0x82][0xf0][0x5]t[0x2][0xb2]Dt"AB[0x18][0x99]r[0x1d][0x90][0xdf][0xde]<[0xae][0x0][0xd0][0x84]&d[0x87][0xc3]3A[0xe1][0x9b]Oz[0xb6]uwc[0xd1][0xcc]#[0x96][0xaa][0xe3][0xb1]![0x8f]T[0xdd][0xac]?[0x87][0xce][0x80][0xd3][0xab]C[0xa7]q[0x1c][0xb2][0x96][0x96][0xa9][0x17][0xb4][0xf0]{[0x83]([0xfe][0x95][0xa9]^c[0xf7][0xc9][0xce][\r][0xfc][0xcf][0xb9][0xdc][0xc2][0x1f]:([0x9d]j[0xbb]v,F[0x8d]9[0xc8][0x16][0x9a][0x15][0x5][0x1e][0x81][0xc8]w[0xb0][0xca][0x2]p[0xee][0xa1][0xb5][0xbf]w[0xb3][0xbf]gex[0x98][0x92]{[0x80][0xbc]{x[0xb3]{8[0x7][0xd9][0x94][0xcc]C[0x6][0xdd][0xc4][0x87]L[0xe9]E[0xc0][0xb2][0xef]0[0xc1]VG[0x95][0x9b][0xaa][0xb1][0x9b][0xc1][0xfa][0xb6][0x8a][0xe]i[0xc2][1[0x17][0x82][0xf9]-M[0x7]m5[0x19][0xa1][0xf6][0x9]e[0xd2]k[0xfc][0xc3]{[0xb2][0xeb][0x1d][0xc][0xe7]a[0xaa][0x0]t[0x9e][0x97]j[0xb2][0xc][0xae]{h[0xc3][0x9f][0x8a][0xd6][0x8c][0x14]R[0x89]{[0xa0][0xe8][0x2][0xa0][0xce][0x90][0x8f]J[0xea][0xae][\n]”

Thanks Shinsuke

6

fess-crawler.log for Postgraduate Module Open Options 2021/22 | SOAS University of London

2021-04-11 23:31:26,381 [WebFsCrawler] INFO Target URL: Postgraduate Module Open Options 2021/22 | SOAS University of London
2021-04-11 23:31:26,381 [WebFsCrawler] INFO Included URL: https://www.soas.ac.uk/students/modulesignup/pg-open-options/.*
2021-04-11 23:31:26,381 [WebFsCrawler] INFO Included URL: (?i).(pdf|PDF)
2021-04-11 23:31:26,382 [WebFsCrawler] INFO Included URL: (?i)..[pP][dD][fF]
2021-04-11 23:31:26,390 [WebFsCrawler] INFO Excluded URL from failures: \Qhttps://www.soas.ac.uk/students/modulesignup/pg-open-options/\E
2021-04-11 23:31:26,391 [WebFsCrawler] DEBUG Crawling Postgraduate Module Open Options 2021/22 | SOAS University of London
2021-04-11 23:31:26,408 [IndexUpdater] DEBUG Starting indexUpdater.
2021-04-11 23:31:26,464 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Queued URL: [UrlQueueImpl [id=rfetgmMBl8hq3_2d5gl8-1.aHR0cHM6Ly93d3cuc29hcy5hYy51ay9zdHVkZW50cy9tb2R1bGVzaWdudXAvcGctb3Blbi1vcHRpb25zLw, sessionId=rfetgmMBl8hq3_2d5gl8-1, method=GET, url=https://www.soas.ac.uk/students/modulesignup/pg-open-options/, encoding=null, parentUrl=null, depth=0, lastModified=0, createTime=1618180286350]]
2021-04-11 23:31:26,491 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Postgraduate Module Open Options 2021/22 | SOAS University of London is not a target url. (0)
2021-04-11 23:31:36,408 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:31:36,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:31:36,417 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:31:36,417 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:31:36,993 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (1)
2021-04-11 23:31:46,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:31:46,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:31:46,412 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:31:46,412 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:31:47,495 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (2)
2021-04-11 23:31:56,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:31:56,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:31:56,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:31:56,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:31:57,997 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (3)
2021-04-11 23:32:06,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:06,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:06,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:32:06,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:08,499 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (4)
2021-04-11 23:32:16,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:16,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:16,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:32:16,412 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:19,002 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (5)
2021-04-11 23:32:26,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:26,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:26,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 338MB, heap 1GB, max 1GB})
2021-04-11 23:32:26,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:26,424 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:1383804928,“total”:12430614528},“swap_space”:{“free”:6511763456,“total”:6874460160}},“cpu”:{“percent”:3},“load_averages”:[0.2, 0.11, 0.08]},“process”:{“file_descriptor”:{“open”:292,“max”:65535},“cpu”:{“percent”:0,“total”:22910},“virtual_memory”:{“total”:4093710336}},“jvm”:{“memory”:{“heap”:{“used”:350515296,“committed”:1173094400,“max”:1173094400,“percent”:29},“non_heap”:{“used”:81712320,“committed”:86175744}},“pools”:{“direct”:{“count”:26,“used”:135274497,“capacity”:135274496},“mapped”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:2,“time”:169},“old”:{“count”:1,“time”:76}},“threads”:{“count”:40,“peak”:40},“classes”:{“loaded”:10266,“total_loaded”:10266,“unloaded”:0},“uptime”:75092},“elasticsearch”:{“nodes”:{“TRpYmpykSxSnrDpM3qsPrQ”:{“timestamp”:1618180346189,“name”:“master-data-node-1”,“transport_address”:“127.0.0.1:9300”,“host”:“127.0.0.1”,“ip”:“127.0.0.1:9300”,“roles”:[“master”,“data”,“ingest”],“indices”:{“docs”:{“count”:19143930,“deleted”:6996},“store”:{“size_in_bytes”:4492337313},“indexing”:{“index_total”:1492274,“index_time_in_millis”:1174300,“index_current”:0,“index_failed”:99,“delete_total”:1506785,“delete_time_in_millis”:22959,“delete_current”:0,“noop_update_total”:0,“is_throttled”:false,“throttle_time_in_millis”:0},“get”:{“total”:1397321,“time_in_millis”:49687,“exists_total”:1250571,“exists_time_in_millis”:47131,“missing_total”:146750,“missing_time_in_millis”:2556,“current”:0},“search”:{“open_contexts”:15,“query_total”:4433299,“query_time_in_millis”:523374,“query_current”:0,“fetch_total”:201679,“fetch_time_in_millis”:155520,“fetch_current”:0,“scroll_total”:89888,“scroll_time_in_millis”:8133796822,“scroll_current”:15,“suggest_total”:0,“suggest_time_in_millis”:0,“suggest_current”:0},“merges”:{“current”:0,“current_docs”:0,“current_size_in_bytes”:0,“total”:9644,“total_time_in_millis”:538127,“total_docs”:25105935,“total_size_in_bytes”:12574770804,“total_stopped_time_in_millis”:0,“total_throttled_time_in_millis”:0,“total_auto_throttle_in_bytes”:4131389440},“refresh”:{“total”:663078,“total_time_in_millis”:935154,“listeners”:0},“flush”:{“total”:20387,“total_time_in_millis”:159900},“warmer”:{“current”:0,“total”:118878,“total_time_in_millis”:1553},“query_cache”:{“memory_size_in_bytes”:30950,“total_count”:1405,“hit_count”:1036,“miss_count”:369,“cache_size”:20,“cache_count”:80,“evictions”:60},“fielddata”:{“memory_size_in_bytes”:136144,“evictions”:0},“completion”:{“size_in_bytes”:0},“segments”:{“count”:445,“memory_in_bytes”:11717367,“terms_memory_in_bytes”:9832226,“stored_fields_memory_in_bytes”:1424320,“term_vectors_memory_in_bytes”:11392,“norms_memory_in_bytes”:16704,“points_memory_in_bytes”:235217,“doc_values_memory_in_bytes”:197508,“index_writer_memory_in_bytes”:0,“version_map_memory_in_bytes”:229,“fixed_bit_set_memory_in_bytes”:0,“max_unsafe_auto_id_timestamp”:-1,“file_sizes”:{}},“translog”:{“operations”:4807,“size_in_bytes”:57003890,“uncommitted_operations”:13,“uncommitted_size_in_bytes”:10407},“request_cache”:{“memory_size_in_bytes”:1705056,“evictions”:0,“hit_count”:1581245,“miss_count”:1503030},“recovery”:{“current_as_source”:0,“current_as_target”:0,“throttle_time_in_millis”:0}},“os”:{“timestamp”:1618180346361,“cpu”:{“percent”:0,“load_average”:{“1m”:0.2,“5m”:0.11,“15m”:0.08}},“mem”:{“total_in_bytes”:12430614528,“free_in_bytes”:1382567936,“used_in_bytes”:11048046592,“free_percent”:11,“used_percent”:89},“swap”:{“total_in_bytes”:6874460160,“free_in_bytes”:6511763456,“used_in_bytes”:362696704},“cgroup”:{“cpuacct”:{“control_group”:"/",“usage_nanos”:149380652700038},“cpu”:{“control_group”:"/",“cfs_period_micros”:100000,“cfs_quota_micros”:-1,“stat”:{“number_of_elapsed_periods”:0,“number_of_times_throttled”:0,“time_throttled_nanos”:0}},“memory”:{“control_group”:"/",“limit_in_bytes”:“9223372036854771712”,“usage_in_bytes”:“9981755392”}}},“process”:{“timestamp”:1618180346361,“open_file_descriptors”:1059,“max_file_descriptors”:65536,“cpu”:{“percent”:0,“total_in_millis”:9139140},“mem”:{“total_virtual_in_bytes”:12778676224}},“jvm”:{“timestamp”:1618180346362,“uptime_in_millis”:474020470,“mem”:{“heap_used_in_bytes”:1694947144,“heap_used_percent”:39,“heap_committed_in_bytes”:4260102144,“heap_max_in_bytes”:4260102144,“non_heap_used_in_bytes”:149709040,“non_heap_committed_in_bytes”:157642752,“pools”:{“young”:{“used_in_bytes”:130256160,“max_in_bytes”:279183360,“peak_used_in_bytes”:279183360,“peak_max_in_bytes”:279183360},“survivor”:{“used_in_bytes”:4667656,“max_in_bytes”:34865152,“peak_used_in_bytes”:34865152,“peak_max_in_bytes”:34865152},“old”:{“used_in_bytes”:1560023328,“max_in_bytes”:3946053632,“peak_used_in_bytes”:2973950728,“peak_max_in_bytes”:3946053632}}},“threads”:{“count”:67,“peak_count”:72},“gc”:{“collectors”:{“young”:{“collection_count”:8998,“collection_time_in_millis”:138825},“old”:{“collection_count”:4,“collection_time_in_millis”:285}}},“buffer_pools”:{“direct”:{“count”:107,“used_in_bytes”:139827002,“total_capacity_in_bytes”:139827001},“mapped”:{“count”:780,“used_in_bytes”:4483412444,“total_capacity_in_bytes”:4483412444}},“classes”:{“current_loaded_count”:13070,“total_loaded_count”:13225,“total_unloaded_count”:155}},“thread_pool”:{“bulk”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:117854},“fetch_shard_started”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:8,“completed”:172},“fetch_shard_store”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“flush”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:26326},“force_merge”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“generic”:{“threads”:5,“queue”:0,“active”:0,“rejected”:0,“largest”:5,“completed”:866196},“get”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:1394919},“index”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:52679},“listener”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“management”:{“threads”:4,“queue”:0,“active”:1,“rejected”:0,“largest”:4,“completed”:64848},“refresh”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:23315644},“search”:{“threads”:7,“queue”:0,“active”:0,“rejected”:0,“largest”:7,“completed”:5552954},“snapshot”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“warmer”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0}},“fs”:{“timestamp”:1618180346363,“total”:{“total_in_bytes”:178636283904,“free_in_bytes”:160689635328,“available_in_bytes”:160689635328},“least_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689635328,“used_disk_percent”:10.046474424896019},“most_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689635328,“used_disk_percent”:10.046474424896019},“data”:[{“path”:"/var/lib/elasticsearch/nodes/0",“mount”:"/ (rootfs)",“type”:“rootfs”,“total_in_bytes”:178636283904,“free_in_bytes”:160689635328,“available_in_bytes”:160689635328}],“io_stats”:{“devices”:[{“device_name”:“dm-0”,“operations”:1433778,“read_operations”:136729,“write_operations”:1297049,“read_kilobytes”:4487412,“write_kilobytes”:29663235}],“total”:{“operations”:1433778,“read_operations”:136729,“write_operations”:1297049,“read_kilobytes”:4487412,“write_kilobytes”:29663235}}},“transport”:{“server_open”:39,“rx_count”:2623004,“rx_size_in_bytes”:3215730451,“tx_count”:2623000,“tx_size_in_bytes”:4396458452}}}},“timestamp”:1618180346424}
2021-04-11 23:32:29,504 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (6)
2021-04-11 23:32:36,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:36,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:36,412 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:32:36,412 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:40,006 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (7)
2021-04-11 23:32:46,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:46,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:46,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:32:46,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:50,508 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (8)
2021-04-11 23:32:56,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:56,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:56,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:32:56,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:01,010 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (9)
2021-04-11 23:33:06,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:06,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:06,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:33:06,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:11,512 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (10)
2021-04-11 23:33:16,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:16,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:16,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:33:16,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:22,014 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (11)
2021-04-11 23:33:26,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:26,410 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:26,411 [IndexUpdater] INFO Processing no docs (Doc:{access 1ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:33:26,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:26,610 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:1380499456,“total”:12430614528},“swap_space”:{“free”:6511763456,“total”:6874460160}},“cpu”:{“percent”:0},“load_averages”:[0.07, 0.09, 0.07]},“process”:{“file_descriptor”:{“open”:292,“max”:65535},“cpu”:{“percent”:0,“total”:23680},“virtual_memory”:{“total”:4093710336}},“jvm”:{“memory”:{“heap”:{“used”:357766696,“committed”:1173094400,“max”:1173094400,“percent”:30},“non_heap”:{“used”:83091712,“committed”:87224320}},“pools”:{“direct”:{“count”:26,“used”:135274497,“capacity”:135274496},“mapped”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:2,“time”:169},“old”:{“count”:1,“time”:76}},“threads”:{“count”:40,“peak”:40},“classes”:{“loaded”:10425,“total_loaded”:10425,“unloaded”:0},“uptime”:135365},“elasticsearch”:{“nodes”:{“TRpYmpykSxSnrDpM3qsPrQ”:{“timestamp”:1618180406436,“name”:“master-data-node-1”,“transport_address”:“127.0.0.1:9300”,“host”:“127.0.0.1”,“ip”:“127.0.0.1:9300”,“roles”:[“master”,“data”,“ingest”],“indices”:{“docs”:{“count”:19143930,“deleted”:6996},“store”:{“size_in_bytes”:4492337313},“indexing”:{“index_total”:1492274,“index_time_in_millis”:1174300,“index_current”:0,“index_failed”:99,“delete_total”:1506785,“delete_time_in_millis”:22959,“delete_current”:0,“noop_update_total”:0,“is_throttled”:false,“throttle_time_in_millis”:0},“get”:{“total”:1397321,“time_in_millis”:49687,“exists_total”:1250571,“exists_time_in_millis”:47131,“missing_total”:146750,“missing_time_in_millis”:2556,“current”:0},“search”:{“open_contexts”:10,“query_total”:4433390,“query_time_in_millis”:523375,“query_current”:0,“fetch_total”:201685,“fetch_time_in_millis”:155520,“fetch_current”:0,“scroll_total”:89903,“scroll_time_in_millis”:8135287623,“scroll_current”:10,“suggest_total”:0,“suggest_time_in_millis”:0,“suggest_current”:0},“merges”:{“current”:0,“current_docs”:0,“current_size_in_bytes”:0,“total”:9644,“total_time_in_millis”:538127,“total_docs”:25105935,“total_size_in_bytes”:12574770804,“total_stopped_time_in_millis”:0,“total_throttled_time_in_millis”:0,“total_auto_throttle_in_bytes”:4131389440},“refresh”:{“total”:663078,“total_time_in_millis”:935154,“listeners”:0},“flush”:{“total”:20387,“total_time_in_millis”:159900},“warmer”:{“current”:0,“total”:118878,“total_time_in_millis”:1553},“query_cache”:{“memory_size_in_bytes”:30950,“total_count”:1405,“hit_count”:1036,“miss_count”:369,“cache_size”:20,“cache_count”:80,“evictions”:60},“fielddata”:{“memory_size_in_bytes”:136144,“evictions”:0},“completion”:{“size_in_bytes”:0},“segments”:{“count”:445,“memory_in_bytes”:11717367,“terms_memory_in_bytes”:9832226,“stored_fields_memory_in_bytes”:1424320,“term_vectors_memory_in_bytes”:11392,“norms_memory_in_bytes”:16704,“points_memory_in_bytes”:235217,“doc_values_memory_in_bytes”:197508,“index_writer_memory_in_bytes”:0,“version_map_memory_in_bytes”:229,“fixed_bit_set_memory_in_bytes”:0,“max_unsafe_auto_id_timestamp”:-1,“file_sizes”:{}},“translog”:{“operations”:4801,“size_in_bytes”:56999549,“uncommitted_operations”:13,“uncommitted_size_in_bytes”:10407},“request_cache”:{“memory_size_in_bytes”:1705056,“evictions”:0,“hit_count”:1581245,“miss_count”:1503030},“recovery”:{“current_as_source”:0,“current_as_target”:0,“throttle_time_in_millis”:0}},“os”:{“timestamp”:1618180406592,“cpu”:{“percent”:0,“load_average”:{“1m”:0.07,“5m”:0.09,“15m”:0.07}},“mem”:{“total_in_bytes”:12430614528,“free_in_bytes”:1380515840,“used_in_bytes”:11050098688,“free_percent”:11,“used_percent”:89},“swap”:{“total_in_bytes”:6874460160,“free_in_bytes”:6511763456,“used_in_bytes”:362696704},“cgroup”:{“cpuacct”:{“control_group”:"/",“usage_nanos”:149382755682171},“cpu”:{“control_group”:"/",“cfs_period_micros”:100000,“cfs_quota_micros”:-1,“stat”:{“number_of_elapsed_periods”:0,“number_of_times_throttled”:0,“time_throttled_nanos”:0}},“memory”:{“control_group”:"/",“limit_in_bytes”:“9223372036854771712”,“usage_in_bytes”:“9983299584”}}},“process”:{“timestamp”:1618180406592,“open_file_descriptors”:1056,“max_file_descriptors”:65536,“cpu”:{“percent”:0,“total_in_millis”:9139920},“mem”:{“total_virtual_in_bytes”:12778676224}},“jvm”:{“timestamp”:1618180406593,“uptime_in_millis”:474080701,“mem”:{“heap_used_in_bytes”:1724618960,“heap_used_percent”:40,“heap_committed_in_bytes”:4260102144,“heap_max_in_bytes”:4260102144,“non_heap_used_in_bytes”:149709040,“non_heap_committed_in_bytes”:157642752,“pools”:{“young”:{“used_in_bytes”:159927976,“max_in_bytes”:279183360,“peak_used_in_bytes”:279183360,“peak_max_in_bytes”:279183360},“survivor”:{“used_in_bytes”:4667656,“max_in_bytes”:34865152,“peak_used_in_bytes”:34865152,“peak_max_in_bytes”:34865152},“old”:{“used_in_bytes”:1560023328,“max_in_bytes”:3946053632,“peak_used_in_bytes”:2973950728,“peak_max_in_bytes”:3946053632}}},“threads”:{“count”:67,“peak_count”:72},“gc”:{“collectors”:{“young”:{“collection_count”:8998,“collection_time_in_millis”:138825},“old”:{“collection_count”:4,“collection_time_in_millis”:285}}},“buffer_pools”:{“direct”:{“count”:107,“used_in_bytes”:139827002,“total_capacity_in_bytes”:139827001},“mapped”:{“count”:780,“used_in_bytes”:4483412444,“total_capacity_in_bytes”:4483412444}},“classes”:{“current_loaded_count”:13070,“total_loaded_count”:13225,“total_unloaded_count”:155}},“thread_pool”:{“bulk”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:117854},“fetch_shard_started”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:8,“completed”:172},“fetch_shard_store”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“flush”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:26326},“force_merge”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“generic”:{“threads”:5,“queue”:0,“active”:0,“rejected”:0,“largest”:5,“completed”:866329},“get”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:1394919},“index”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:52679},“listener”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“management”:{“threads”:4,“queue”:0,“active”:1,“rejected”:0,“largest”:4,“completed”:64857},“refresh”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:23318531},“search”:{“threads”:7,“queue”:0,“active”:0,“rejected”:0,“largest”:7,“completed”:5553074},“snapshot”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“warmer”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0}},“fs”:{“timestamp”:1618180406594,“total”:{“total_in_bytes”:178636283904,“free_in_bytes”:160689659904,“available_in_bytes”:160689659904},“least_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689659904,“used_disk_percent”:10.046460667332624},“most_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689659904,“used_disk_percent”:10.046460667332624},“data”:[{“path”:"/var/lib/elasticsearch/nodes/0",“mount”:"/ (rootfs)",“type”:“rootfs”,“total_in_bytes”:178636283904,“free_in_bytes”:160689659904,“available_in_bytes”:160689659904}],“io_stats”:{“devices”:[{“device_name”:“dm-0”,“operations”:1433860,“read_operations”:136729,“write_operations”:1297131,“read_kilobytes”:4487412,“write_kilobytes”:29663744}],“total”:{“operations”:1433860,“read_operations”:136729,“write_operations”:1297131,“read_kilobytes”:4487412,“write_kilobytes”:29663744}}},“transport”:{“server_open”:39,“rx_count”:2623100,“rx_size_in_bytes”:3215739314,“tx_count”:2623096,“tx_size_in_bytes”:4396592047}}}},“timestamp”:1618180406610}
2021-04-11 23:33:32,516 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (12)
2021-04-11 23:33:36,410 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:36,411 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:36,413 [IndexUpdater] INFO Processing no docs (Doc:{access 1ms}, Mem:{used 342MB, heap 1GB, max 1GB})
2021-04-11 23:33:36,413 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:43,018 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (13)
2021-04-11 23:33:46,411 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:46,411 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.

7

fess-crawler.log for https://www.soas.ac.uk/util/eventscrawler/ - nothing crawled

2021-04-11 23:46:59,658 [WebFsCrawler] INFO Target URL: https://www.soas.ac.uk/util/eventscrawler/
2021-04-11 23:46:59,658 [WebFsCrawler] INFO Included URL: https://www.soas.ac.uk/util/eventscrawler/*
2021-04-11 23:46:59,658 [WebFsCrawler] INFO Excluded URL: .
2021-04-11 23:46:59,668 [WebFsCrawler] DEBUG Crawling https://www.soas.ac.uk/util/eventscrawler/
2021-04-11 23:46:59,689 [IndexUpdater] DEBUG Starting indexUpdater.
2021-04-11 23:46:59,759 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Queued URL: [UrlQueueImpl [id=AVyh6-FfHWyes_9AjAu9-1.aHR0cHM6Ly93d3cuc29hcy5hYy51ay91dGlsL2V2ZW50c2NyYXdsZXIv, sessionId=AVyh6-FfHWyes_9AjAu9-1, method=GET, url=https://www.soas.ac.uk/util/eventscrawler/, encoding=null, parentUrl=null, depth=0, lastModified=0, createTime=1618181219620]]
2021-04-11 23:46:59,766 [Crawler-AVyh6-FfHWyes_9AjAu9-1-3] DEBUG The url is null. (0)
2021-04-11 23:46:59,772 [Crawler-AVyh6-FfHWyes_9AjAu9-1-6] DEBUG The url is null. (0)
2021-04-11 23:46:59,774 [Crawler-AVyh6-FfHWyes_9AjAu9-1-4] DEBUG The url is null. (0)
2021-04-11 23:46:59,777 [Crawler-AVyh6-FfHWyes_9AjAu9-1-7] DEBUG The url is null. (0)
2021-04-11 23:46:59,778 [Crawler-AVyh6-FfHWyes_9AjAu9-1-8] DEBUG The url is null. (0)
2021-04-11 23:46:59,780 [Crawler-AVyh6-FfHWyes_9AjAu9-1-5] DEBUG The url is null. (0)
2021-04-11 23:46:59,782 [Crawler-AVyh6-FfHWyes_9AjAu9-1-2] DEBUG The url is null. (0)
2021-04-11 23:46:59,787 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] INFO Crawling URL: https://www.soas.ac.uk/util/eventscrawler/
2021-04-11 23:46:59,789 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Searching indexed document: https:%2F%2Fwww.soas.ac.uk%2Futil%2Feventscrawler%2F;role=Rguest
2021-04-11 23:46:59,792 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Query DSL:
2021-04-11 23:46:59,801 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Query DSL:
2021-04-11 23:46:59,803 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Getting the content from URL: https://www.soas.ac.uk/util/eventscrawler/
2021-04-11 23:46:59,812 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Initializing org.codelibs.fess.crawler.client.http.HcHttpClient
2021-04-11 23:46:59,859 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Accessing https://www.soas.ac.uk/util/eventscrawler/
2021-04-11 23:46:59,859 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] INFO Checking URL: https://www.soas.ac.uk/robots.txt
2021-04-11 23:46:59,871 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG CookieSpec selected: default
2021-04-11 23:46:59,887 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection request: [route: {s}->https://www.soas.ac.uk:443][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 200]
2021-04-11 23:46:59,907 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection leased: [id: 0][route: {s}->https://www.soas.ac.uk:443][total kept alive: 0; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 23:46:59,909 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Opening connection {s}->https://www.soas.ac.uk:443
2021-04-11 23:46:59,913 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connecting to www.soas.ac.uk/212.219.139.54:443
2021-04-11 23:46:59,913 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connecting socket to www.soas.ac.uk/212.219.139.54:443 with timeout 0
2021-04-11 23:46:59,953 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
2021-04-11 23:46:59,953 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2021-04-11 23:46:59,953 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Starting handshake
2021-04-11 23:47:00,059 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Secure session established
2021-04-11 23:47:00,059 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG negotiated protocol: TLSv1.2
2021-04-11 23:47:00,059 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2021-04-11 23:47:00,060 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG peer principal: CN=www.soas.ac.uk, O=School of Oriental and African Studies, L=London, ST=“London, City of”, C=GB, SERIALNUMBER=1916-05-23, OID.2.5.4.15=Government Entity, OID.1.3.6.1.4.1.311.60.2.1.3=GB
2021-04-11 23:47:00,060 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG peer alternative names: [www.soas.ac.uk, soas.ac.uk]
2021-04-11 23:47:00,060 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG issuer principal: CN=QuoVadis Europe EV SSL CA G1, O=QuoVadis Trustlink B.V., C=NL
2021-04-11 23:47:00,063 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection established 212.219.238.219:41542<->212.219.139.54:443
2021-04-11 23:47:00,063 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Executing request GET /robots.txt HTTP/1.1
2021-04-11 23:47:00,063 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Target auth state: UNCHALLENGED
2021-04-11 23:47:00,064 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Proxy auth state: UNCHALLENGED
2021-04-11 23:47:00,065 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> GET /robots.txt HTTP/1.1
2021-04-11 23:47:00,065 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> Host: www.soas.ac.uk
2021-04-11 23:47:00,065 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> Connection: Keep-Alive
2021-04-11 23:47:00,065 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> User-Agent: Mozilla/5.0 (compatible; Fess:12.0; +SOAS Events)
2021-04-11 23:47:00,065 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> Accept-Encoding: gzip,deflate
2021-04-11 23:47:00,066 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “GET /robots.txt HTTP/1.1[\r][\n]”
2021-04-11 23:47:00,066 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “Host: www.soas.ac.uk[\r][\n]”
2021-04-11 23:47:00,066 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]”
2021-04-11 23:47:00,066 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “User-Agent: Mozilla/5.0 (compatible; Fess:12.0; +SOAS Events)[\r][\n]”
2021-04-11 23:47:00,066 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “Accept-Encoding: gzip,deflate[\r][\n]”
2021-04-11 23:47:00,066 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “HTTP/1.1 301 Moved Permanently[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Server: nginx[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Date: Sun, 11 Apr 2021 22:47:00 GMT[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Content-Type: text/html[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Content-Length: 162[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Connection: keep-alive[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Location: https://www.soas.ac.uk/static/robots.txt[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Strict-Transport-Security: max-age=0; includeSubdomains; [\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “X-Frame-Options: SAMEORIGIN[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “301 Moved Permanently[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “

301 Moved Permanently

[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “
nginx[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 23:47:00,067 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << HTTP/1.1 301 Moved Permanently
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << Server: nginx
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << Date: Sun, 11 Apr 2021 22:47:00 GMT
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << Content-Type: text/html
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << Content-Length: 162
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << Connection: keep-alive
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << Location: https://www.soas.ac.uk/static/robots.txt
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << Strict-Transport-Security: max-age=0; includeSubdomains;
2021-04-11 23:47:00,070 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << X-Frame-Options: SAMEORIGIN
2021-04-11 23:47:00,075 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection can be kept alive indefinitely
2021-04-11 23:47:00,078 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection [id: 0][route: {s}->https://www.soas.ac.uk:443] can be kept alive indefinitely
2021-04-11 23:47:00,078 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0: set socket timeout to 0
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection released: [id: 0][route: {s}->https://www.soas.ac.uk:443][total kept alive: 1; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG CookieSpec selected: default
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection request: [route: {s}->https://www.soas.ac.uk:443][total kept alive: 1; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Connection leased: [id: 0][route: {s}->https://www.soas.ac.uk:443][total kept alive: 0; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0: set socket timeout to 0
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Executing request GET /util/eventscrawler/ HTTP/1.1
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Target auth state: UNCHALLENGED
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG Proxy auth state: UNCHALLENGED
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> GET /util/eventscrawler/ HTTP/1.1
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> Host: www.soas.ac.uk
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> Connection: Keep-Alive
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> User-Agent: Mozilla/5.0 (compatible; Fess:12.0; +SOAS Events)
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> Accept-Encoding: gzip,deflate
2021-04-11 23:47:00,079 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “GET /util/eventscrawler/ HTTP/1.1[\r][\n]”
2021-04-11 23:47:00,080 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “Host: www.soas.ac.uk[\r][\n]”
2021-04-11 23:47:00,080 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]”
2021-04-11 23:47:00,080 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “User-Agent: Mozilla/5.0 (compatible; Fess:12.0; +SOAS Events)[\r][\n]”
2021-04-11 23:47:00,080 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “Accept-Encoding: gzip,deflate[\r][\n]”
2021-04-11 23:47:00,080 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 >> “[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “HTTP/1.1 404 Not Found[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Server: nginx[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Date: Sun, 11 Apr 2021 22:47:00 GMT[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Content-Type: text/html; charset=UTF-8[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Transfer-Encoding: chunked[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Connection: keep-alive[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Vary: Accept-Encoding[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Cache-Control: no-cache, private[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “Content-Encoding: gzip[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “23c7[\r][\n]”
2021-04-11 23:47:00,104 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x3][0xbd][0xdb]n$[0xc9]q}[0xd7]W[0x94])[0x18][0xf0][0xc2]S=[0x95][0xf7][0xcc][0x99][0xdd][0x85]mY[0x82][0x1f][0xbc][0xb0][0x1][0xdb][0x2][0xc][0xc3][0xf]5[0xcd][0xde]![0xbd]M6A[0xf6]\v[0x4][0xfd][0xbb][0xcf]9[0x91][0xd9]M[0xe][0xb9][0x92][0x9e]${[0xb3][0x9a]u[0xcb][0xcc][0xc8][0xc8][0x88][0x13]’[0xa2][0xe6][0xdb][0xbf][0xf9][0xe7][0xfb][0xcd][0xfe][0xf7][0xbf][0xff]v[0xba]:[0xde][0xec][0xbf][0xff][0xd5][0xb7]v[0x98][0xf0][0xbf]o[0xaf]v[0xeb][0xa5][0xfd][0xd4][0x9f]7[0xbb][0xe3]:m[0xaf][0xd6][0xfb][0x87][0xdd][0xf1][0xbb][0x8b][0xff][0xfa][0xcf][0xdf][0xcd][0xf5]bz[0xfd][0xf5][\r][0xb7][0xeb][0xcd][0xee][0xbb][0x8b][0xfb][0xc3][0xbb][0xc3][0xf1][0xe1]b[0xda][0x1e]n[0x8f][0xbb][[0xdc]~{[0xb8][0xbe][0xbd][0xdc]}~u{[0xf8][0xf1][0xb0][0xdf][0x1f]>=}[0xf2][0xe1][0xf8][0xf3]~[0xf7][0xfd][0xf4][0xe8][0xef][0xe][0x97]?O[0x98][0xde][0xad][0xdb][0x9f][0xde][0xdf][0x1f]>[0xdc]^[0xce][0xdb][0xc3][0xfe]p[0xff]f[0xfa][0xf5][0xef][0x1a][0xff][0xef][0xed]4[0xfe][0xf6][0xde][0xbf][0x9d]~D?o&[0x17][0xef]>[0xbf]v[0x9b]8[0xfd][0xcb]n[0xff]qw[0xbc][0xde][0xae][0xaf][0xa6][0xbc][0xbf]^[0xf7][0xaf][0xa6][0x87][0xf5][0xf6]a~[0xd8][0xdd][0xff][0xf8]v[0xba]Y[0xef][0xdf][0xdf][0xbe][0x99][0x96][0xb7][0xd3][0xdd]zyy}[0xfb]~[0xc6]h[0x8f][0x87][0x9b]7SLw[0x9f][0xdf]N[0xfc][0xd5][0xaf][0x1e][0x8f]e[0xc5]@[0xb6][0x1f][0xee][0x1f][0xd8][0xdd][0x1d][0xe6]q[0xdc][0xdd][0xbf][0x9d][0x8e][0xbb][0xcf][0xc7][0xf9]r[0xb7]=[0xdc][0xaf][0xc7][0xeb][0x3][0xde]v{[0xb8][0xdd][0xf1][0xd1]'O[0xbe][0xb9]:|[0xdc][0xdd][0xe3][0xf9]g[0xb7]cJ[0xbb][0xfb][0xfd][0xf5][0xb][0xcf][0xbc]{w[0xff]?[0xc7][0xeb][0xe3]~[0xf7][0xbf][0x14][0xc0][0xe1][0x1e][0xf7][0x9d][0xc6]g[0x9d][0x8c][0xc1][0xed][0xf6]wb$O^[0xbb]=[0xee]^Mw[0xf7];[0xbc][0xb4]K+HZi[0xfa][0xcd][0xe1][0xf6][0xe1][0xb0][0x1f]^M?[0x1c]n[0xd7][0xed][0x1][0xc7][0xdd][0xed][0x1e][0x87][0x8b][0xff]z[0xf7][0xe1][0xf6][0xf8][0x81][0xa7][0xf][0x17][0xf8][0xf3][0xaf][0xdf][0xed][0xac][0x93]q[0xea][0x6][0x87][0x87][0xbb]u[0xfb][0xbc][0xbf][0xe3][0xfa]n[0x8f][0xe][0x8f][0xf7][0xf8][0xef][\n]”
2021-04-11 23:47:00,106 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[0xff]]>YM[0xae][0xe3][0xef]~[0xf7]vL[0xf]k[0xb9][0xef][0x1e]vo[0xa6][0xf1][0xeb][0xed][0x4][0xb9]q[0x5][0xf7][0xf3][0xba][0xbf]~[0x8f]Y[0x1d][0xf]w[0x8b]J}[0xfc][0x89][0xf7]b[0x8e]w[0x9f]’[0xcc][0xed][0xfa]r[0xfa][0xf5]o[0x17][0xfe][0x1f]/}[0x9e][0x1f][0xae][0xd6][0xcb][0xc3]'h[0x0].[0xf3]?[0xde]v[0xff][0xfe][0xdd][0xfa]w[0xce][0xd7]W[0xd3][0xb9][0xd9][0xf8]o[0xce][0xea][0xe2]v7T[0x99]O[0xd7][0x97][0xc7]+[0xfc][0xb5],[0xfb][0xf2]x4][0xfb]uyZ[0xc1]7[0xcf][0x87][0xa1][0x99][0x8f][0xb7]q 'u|3U[0xe]j1]|[0xa9][0x87][0x97][0xb7][0xc6]x7[0xd7]w[0xfe][0xb4][0xbb]~[0x85]e~w[0xd8]v%[0xe9][0x92][0xdc][0xef]~<>[[0xaf][0xcd][0xd5][0xf5][0xe5][0xe5][0xee]v[0xfe][0xfc]0C4[0xb7][0xe8][0xe0][0xf2][0xfa][0xe1]n[0xbf][0xfe][0xfc][0xb2]fo[0xde][0xed][0xf][0xdb][0x9f][0x1e][0xdf][0xa5][0x13][0xcf]o[0xbb][0xdf][0xad]?[0xcd][0xfb][0x3][0xf6][0xd8]’[0xcc][0xf8][0x1]O[0xcc]7[0xf][0xfa]=[0xeb][0x1a][0x1e][0xd4]-[0xeb]~[0xf][0xd9][0xfe][0xa5][0xa7]y[0xe6][0xed][0x84]9[0xbe][0xfb][0xe9][0xfa]8[0xfd]|w[0xb5][0xbb]}x3[0xad][0x1f][0x8e][0x7][0x9c][0xbe]9|[0xf9][0xfa][0xdc]W~5JI[0xe7][0xe6][0xc3]q[0xc7][0x5][0x1b][0xd2]l[0xad]=[0x97][0x12][\r][0xda][0x8a]][0xcb]=}[0xb3]~>[0xad][0xdf][0xe2]#W[0xeb]dZz?[0xa7][0x5]]&[0x97][0x9e]/[0xe7][0xf9]uo[0xde][0xac]?[0x1e][0xf5][0xd6]n2[0xdf]L[0x17][0x17]o[0xcf][0xd2][0xd5][0xc2]c[0xe7][0xef]w[0xeb]=[0xd7][0xf4]x[0xf5]|l[0xbb][0xcf][0xdb][0xdd][0x1d]w[0xe7][0xfc][0xf0][0xe1][0x6]#[0xf9][0xf9][0xeb][0xad][0xf1]OKt[0xe1][0xb7]o[0xbf]6*[0xfe][0xb4]I[0xb4][0x7][0x96]WS[0xff][0xff][0x8d][0xfb][0xe6]t3[0xb6][0xdf][0xe3][0xed][0xf4][0xf4][0xce]M[0xc0][0x8d]?[0xee]w[0x9f]9[0xd1]1w[0x93][0xc5][0xa9][0x97][0xb0][0xbc]Z[0x1f][0x8d][0xf9]f[0xf7][0xf0][0xb0][0xbe][0xdf][0xcd][0x9f][0xee][0xd7][0xbb];I[0xe2]4y[0xbe][0xf9][0xed]$[0xed][0x9d][0xaf][0x8f][0xbb][0x1b],[0xe3]vgf[0xf8][0x6]=\uM//l[0x98][0xe7][0x1d][0xd0][0xfe][0xe1]}3[0xa4][0x2][0xb][0xe0][0x1e][0xad]QXl[0x17][0xfe][0xb9]7[0xbc]z[0xe9][0xad][0xeb]#[0xcd][0x91][0x9][0xd2]&|[0xb8][0xfe][0x2][0xa3][0xe6][0x1d][0x7][0xf6]dW[0xc6]ey[0xe2][0x88][0xfe]\[0x9f][0x1b]n[0xa1]n[0xba][0xfb][][0xfd][0x8b][0xa6][0xbb]>[0xf7]"_-[0xa3]O[0xe9][0xd5]tn[0x96]M[0xfa][0xe6][0xc9][0xf0][0xaf]o[0xaf][0xe0]8[0x8f][0xa1][0xcf]{i[0x8]'7[0xf8]d '[0xbb][0xf5][0xa3][0xfe][0xf7][0xa7][0x94][0xe3]z[0xbf][0xff][0xf0]p[0xec][0xbe][0xa7]/[0xe0][0xbb][0xf5][0xe1][0x1a][0x9a][0xe0][0x9c]I[0x97][0xa7][0x1e][0xae][0xee][0xaf]o[0x92]@[0x87]R[0xe4]|[0xde][0x97]3-[0xdf][0x9b][0xbe][0x13][0xf]p[[0xd7]Gh[0xd7][0xa6]<[0xef][0x19]]mw[0xd3][0xdf][0x8f][0x1f][0x18]/[0xd0].[0x8]/[0xe9][0x99]n[0x9c][0x9][0x99][0xc6][0xef]-\[0xea][0xc3]#[0xad]8[0x1][0x95]'[0xeb][0xf7][0x82][0xad]&*8)[0xb5][0xdb][0x84][0xa7][0x88][0xe5][0xf0]pm[0xc2][0xbf][0xdf][0xed]![0x8f][0x8f][0xcf]]o[0xef][0xff][0x91][0xbe]?[0xea]4>[0xeb][0xf4][0xf6]p[0xb3][0xee][0xcf][0x9d]l[0x12][0x1d][0x9d])[0xe5]K[0xef][0xfd][0xf1]z[0xbf][0x9b][0xef]V[0xba][0xb9][0xaf][0xcf]<[0xd9][0x5][0x9a][0xef][0x13][0xb9][0x9d][0xba][0xee]c[0x9][0xbf]([0xc7][0xaf]e7,[0xd7][0x8b]7[0x1f][0xbe][0xe3],[0x1f][0xd9]Z[0xff][0x8b]/[0x6]F[0xbd]:\[0xbe][0xf0][0xe6][0x17][0xd6][0xe1][0xc5][0x17]>[0x1f]n|[0x1e][0x8f][0xae][0x94][0xf2][0xb]2[0x1d][0xf8][0xd2][0x14][0xcf][0xbf][0x1][0xff][0xe1]fwy[0xbd]NGC[0xd6][0x9d]IP[0xd0]o[0xa6]?<[0xb9]OC[0xf8]e[0x1f]}}[0xb][0xad][0x8f]O[0x7][0xfd][0xc7]o_[0x1b][0xb6]6D[0xff][0xfa][0xc][0xe9][0xbf]%[0xc2][0xfe][0xfe]Y[0x17][0xba]ty[0xfd]q[0xd2][\n]" 2021-04-11 23:47:00,106 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << "|w[0xf1][0xcc][0xa7]\<[0xe6][0xf1][0xfd]'[0x87][0xf6][0xc2]}[0xbf][0xfc][0xee][0xaf]l[0xff]/<[0xab][0xe7][0xaf][0xdc]x[0xfc][0x19][0xb4]x[0xf6][0xbe][0x8b][0xef][0xff][0xe3]p[0xff]3[0x11][0xe8][0xe]K[0x81][0xbd][0xf0][0xf3][0xe1][0xc3][0xb4][0x2][0x3][0xef][0xf][0x87][0x9f][0xb0].X[0xb4]{[0xac][0xe2][0x87][0xfd]%V[0xec]8[0xbd][0xdb][0xe1][0x4]\[0xe5][0x6][0x82]rb[0xc]/[0xce][0xe1][0x89][0x89]z[0xba]P[0x17][0xdf][0xfb][0xf0][0xf1][0xfd][0xf4][0xf1]z[0xf7][0xe9][0x9f][0xe][0x9f][0xbf][0xbb][0xa0][0x8f]t!O[0xd5]]L[0x9f]o[0xf6][0xb7]x[0xcf][0xd5][0xf1]x[0xf7][0xe6][0xf5][0xeb]O[0x9f]>m>[0x85][0xcd][0xe1][0xfe][0xfd]k[0xbf],[0xcb]k<v1a{[0xed][0xe7][0xfb][0xf]{[0xc4]U[0xbb][0x8f][0xbb][0xdb][0xc3][0xe5]%[0x2][0xab][0xfd][0xf5][0xdd][0xd7][0xe7][0xd0][0xfb][0xe1][0xa7][0xdd]L[0xf3][0xf1][0x8]S[0x18][0x85]a2[0xa7][0xf3]7p[0x9d][0x88]8p[0xf8][0xee][0x2]A[0x92][0x8b][0x18][0x96]6[0xed][0xe5]w[0x17]?4[0xbf][0x9][0xa9]N~[0xd9][0xc4]%[0xac]>l\[0xc6]_v[0xd0]p[0xa7][0xb6]Y[0x96][0x0]sT[0xab][0xe3][0x85]\zkW[0xd3][0xc6][0xb5]2[0x85][0xcd]R[0xda][0xe4][0xe3][0xc6][0xa7]2[0xe]v=lb[0xe2]u[0xdf]2/[0x94][0xe2][0xc6]a<[0xbf]D[0x8f]>B[0xcb][0xdb]M[0xac][0x19][0x1d][0x15][0xbf])[0x95][0xf]bl[0xb5]%[0xdc][0xe2][0xeb][0x1e][0x83][0xa8][0x1b][0x17][0xfd]f[0xf1]8[0xa6][0xba]Y[0x1c][0xce][0xc7][0xfc][0xd1][0x85]M-i[0xbb]q.o[0x96][0x80][0xae][0x3][0xee][0xc9]i[0x13]"[0xee]i[0xb8][0xb7]ap)nB[\r][0xba]'[0x95][0xc2][0xc9][0xa1]/[0x1c]}[0xda][0xb4][0xda]p[0x8f]C[0xbf][0xb1]m<^[0x1a][0x2]_[0x9e]7[0x19][0x17][0x16][0xcf][0x99]/[0xde][0xcd]xG[0xeb]?[0x16][0x8c]9x[0xfc][0xf0]8[0x13]+~[0xc4][0x88][0xf1][0xb5]4o[0xdc][0x16]c[\n]" 2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << "[0xf3][0xc6]c[0x0]1[0xe3][0xe6][0x94][0xfc]&g[0x9c][0xa9][0xc9]m|)3[0xc4]\7[0xc9][0xf3]L([0x9b][0x12][0xf3][0xcc][0xf3]mO1[0xe0]M[0xb5]l[0xd1][0xe6][0x8c][0xc6][0xe1]o[0x17][0xd8][0xa1][0xc7][0x9f][0x90]2;[0xda][0xa3]i[0x8d][0xe7][0xc2][0x9a]6[0x9][0xba]d[0xad][0x84]9or[0x8b][0xe8][0x1c][0x2]M[0x9b][0x1c][o[0xed][0x1a][0xd6][0xb9]DtV#%[0x8a][0x1b]{[0xdb][0x9f][0xac][0x9e][0x17][0x13]N@Dn[0xb4][0xfd][0xa2][0xcf]e[0xe6][0xa2]zK[0xc4]4C[0x8b][0x90][0x1d]FU[\n]"
2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “d[0x16][0x2][0x9e]t[0x14]O[0xc1][0xc8]|[0x81][0xcc]3[0xc6][0x9e]2[0xba]\0[0xea][0xea][0xb0]”[0x1c][0x98][0xcb]X[0x2][0xdf]8[0x99][0x8c]Q[0xe2]R[0x88]n[0xbf][0xe1]S[0xb5]z,!E[0x87]f[0xe3][\n]"
2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << "[0xef][0xae][0x19][0xd7]q[0xcf][0x2][0xed][0x89]|[0xaa]D[0xce][0x15]]e[0xaa].[0x5]*^[0xa4][0xb0]8[0x2][0x1e][0xa1][0x0][0x9e]'[0xfc][0x1e]g*[0x17]'[0x14][0xbc][0xb9][0xe0][0xc6][0x5][0xd7][0x1c][0xa5][0xb8]``[0xb5]R[0x93][0x1e]p[0x6][0xeb][0x8b]E/P[0xe4]%a[0x19][0x1c].,x[0xc4][0xd6];Cg[0x1a]op%s~[0xe8][0x5][0xf7][0xe3][0xb7][0xe3]o[0xbc]1@[0xff]<[0xb5][0x93]K[0xe6])[0xa0][0xc2]I@[0xb9][0x96][0xca][0xb5][0x80]2[0xe1][0xf],;[0x1f][0xc3][0x11][0xf3][0x87][0x1c][\n]" 2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << "W[0x9][0xa2][0xa3][0x8][0x12]nMR0[0x8]u[0xc1][0x88][0xdd][0x12]L[0x82].E[0xce][0x82][0xf3]i[0xd4]p[\n]" 2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << "e[0x81][0xd6][0x16]^C[0xd7]|[0x1c][0xb3][0xc5]}[0x9c]z[0xe5]V[0xa4]jq[0x2]U[0x8e][0x8f]c><[0xb5]s[0xb9][0xcc]_[0x90].[0xa6]L[0x1d][0x83][0xc][0xf8][0x18][0xb7][0x83][0x89][0x10]k[0xe2](0t[0x11][0xb0][0xf5][0xb0][0xeb][0xd1]{[0x8c][0x99][0xdb]mk[0xfb](p)[0xa2][0x8e]x[0x11][0xb7][0x17][0x97]{i[0x5][0xb][0xc3]][0xdf][0xa0][0xd1]q[0x13]9[0xa3][0xa0][0xff]0|[0xac][0xbe][0x8b][0x9a][0xf4]~[0xd3][0xb4][0x9f][0xb8]}[0x9d]v[[0xd0]v[0xe6][0xcc][0x17][0xad][0xc]f[0xe5](6lL[0xf6]H[0xbd][0x87][0x8c]O?[0xe2][0xc2][0x99]WJ%T[0x9c][0x8b][0x89][0x2][0xcf][0x1a]}5[0x8d]\(pN[0x14][0x1b]O[0xba][0xc6][0xe]00N[0x1d][0xcb][0xde]E[0xe0]rW[0x3][0xef][0xd8]3F[0x8c]Q[0xd8]!c$[0x19][0xb3][0x80]Vx[0xda][0x4]lt[0x9a]&[0xae][0x87][0x9][0x16]R[0x82][0x94][0xb1][0xb9]a[0xb2][0xb0][0x93][0x17][0xea][0x11][0xd5][0x9a][0x81]{}[0xa1][0xd6][0xed][0xa9][\n]”
2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “TJN[0x2][0xc3][0xa0][0xd4][0xb0]:[0x10]e[0xf8][0x1c]~s[0xb5][0xd0]i[0x8]X[0x7]H6bP[0x99][0x2]N[0xd2][0xb2]%[0xbb][0xde][0xa5][0xa3]A[0xd9]C[0xda][0x8d][0x16][0xca]W[0xed]@<[0x8a][0xfb]p[0xdd]S9+.[0xa0]C[![0xad]+[0xd7][0x17][0x96],[0x95][0xb0][0xc2][0xc6][0xb4]IM[0xdf][0xcb]nq[0xb0][\n]”
2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[0x89]f[0x6][0xd3][0xa0][0xe6]x[0xac]P[p)[0xe4]D[0xd8]I!9H-[0xd0][0xc]y[0xec][0x1c]jt[0x8c][0xbc]&[0x15][0xce][0xf0][0x17][0xbc]-a[0xdb][0xe7][0xe4]’[0xf]S[0xe1]a[0x1f][0x9a][0x93][0xa1][0xcf]i[0xe5][0x9c][0xda]d[0xad]uKi[0xa0][0xbb][0xca][0xb3]P[0x97][0xde][0xe][0xab][0x15]0[0xcb]%l[0xa9][0xcd][0xb8][0x90]*W[0x16][0x93][0xc4][0x95]J[0xb1][0x9a][0xd5][0xcb]f[0x12])[0x1a]nPN[0x91]b[0xe7][\n]”
2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << “[0xd3][0x87][0xf1][0x12][0x84][0x99][0xbc][0xf6]|[0xb4][0xa5][0xae][0xb0]Z[0x85][*I[0xef]rXi 8~;[0xf4][0x1][0xa4][0x12]1[0xb][0xe7][0xa2][0x86][0xd0]0+[0xa8]T[0xac][0x91][0xd3][0xe4][0xc6][0xae][0xb][0xfd]Rsu[0xa5][0x1e]Oj[0xfa][0xa3][0xd0]C[0xdc][0x9e][0xf9] &[0x18][0xa2][0xc6][0xee]8pZ[0xe3]B[0x99]9[0x89][0xaf][0xd2][0xb8]b#G[0xbe][0xca][0xe1]>X[[0xa9]X[0xe6][0xbe]F[0x0][0x6][0x11]@[0xb8][0x89][0xfa]F[0xa3]I7[0x91][0xf0][0xa4][0xa4]6[0xb3]w[0x8c][0xb][0x8f]A[0xce][0xd0][0xad][0x2][0xe9][0xc3]aQc[0xd0]c[0xab][0x14][0x1b][0xef][0x92]”[0xd3][\n]"
2021-04-11 23:47:00,107 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG http-outgoing-0 << "H{h[0x12][0x83][0xd7]~[0xe5]|p[0x93][0xec]([0x14]h[0xa1][\r][0xf0][0xf2]p[0x10][0xc]

2021-04-11 23:47:28,310 [Crawler-AVyh6-FfHWyes_9AjAu9-1-4] DEBUG The url is null. (19)
2021-04-11 23:47:28,311 [Crawler-AVyh6-FfHWyes_9AjAu9-1-5] DEBUG The url is null. (19)
2021-04-11 23:47:28,313 [Crawler-AVyh6-FfHWyes_9AjAu9-1-2] DEBUG The url is null. (19)
2021-04-11 23:47:29,691 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:47:29,691 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:47:29,691 [Crawler-AVyh6-FfHWyes_9AjAu9-1-1] DEBUG The url is null. (19)
2021-04-11 23:47:29,692 [IndexUpdater] INFO Processing no docs (Doc:{access 1ms}, Mem:{used 346MB, heap 1GB, max 1GB})
2021-04-11 23:47:29,692 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:47:30,404 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2021-04-11 23:47:30,404 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2021-04-11 23:47:31,192 [WebFsCrawler] DEBUG infoMap: WebFsCrawlExecTime=31711 => {CrawlerStartTime=2021-04-11T23:46:59.440+0100, WebFsCrawlStartTime=2021-04-11T23:46:59.461+0100}
2021-04-11 23:47:31,192 [WebFsCrawler] INFO [EXEC TIME] crawling time: 31711ms
2021-04-11 23:47:35,404 [CoreLib-TimeoutManager] DEBUG Closing expired connections
2021-04-11 23:47:35,404 [CoreLib-TimeoutManager] DEBUG Closing connections idle longer than 60000 MILLISECONDS
2021-04-11 23:47:39,691 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:47:39,691 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:47:39,693 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 346MB, heap 1GB, max 1GB})
2021-04-11 23:47:39,693 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:47:39,693 [IndexUpdater] DEBUG Finished indexUpdater.

8

Where is NoRouteToHostException?
Since it’s to hard to read logs, please upload it to somewhere…

9

Sorry Shinsuke,

Here is a cutdown from fess-crawler.log

2021-04-09 21:06:42,292 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG http-outgoing-22: Shutdown connection
2021-04-09 21:06:42,292 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connection discarded
2021-04-09 21:06:42,292 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connection released: [id: 22][route: {s}->https://www.soas.ac.uk:443][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 200]
2021-04-09 21:06:42,292 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] INFO I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://www.soas.ac.uk:443: No route to host (Host unreachable)
2021-04-09 21:06:42,292 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG No route to host (Host unreachable)
java.net.NoRouteToHostException: No route to host (Host unreachable)
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] INFO Retrying request to {s}->https://www.soas.ac.uk:443
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG CookieSpec selected: default
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connection request: [route: {s}->https://www.soas.ac.uk:443][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 200]
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connection leased: [id: 23][route: {s}->https://www.soas.ac.uk:443][total kept alive: 0; route allocated: 1 of 20; total allocated: 1 of 200]
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Opening connection {s}->https://www.soas.ac.uk:443
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connecting to www.soas.ac.uk/212.219.138.54:443
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connecting socket to www.soas.ac.uk/212.219.138.54:443 with timeout 0
2021-04-09 21:06:45,300 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG http-outgoing-23: Shutdown connection
2021-04-09 21:06:45,300 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connection discarded
2021-04-09 21:06:45,300 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connection released: [id: 23][route: {s}->https://www.soas.ac.uk:443][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 200]
2021-04-09 21:06:45,300 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Cancelling request execution
2021-04-09 21:06:45,300 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Failed to access to Postgraduate Module Open Options 2021/22 | SOAS University of London
org.codelibs.fess.crawler.exception.CrawlingAccessException: No route to host(No route to host (Host unreachable)): Postgraduate Module Open Options 2021/22 | SOAS University of London

What is confusing is

2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connecting to www.soas.ac.uk/212.219.138.54:443
2021-04-09 21:06:42,293 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Connecting socket to www.soas.ac.uk/212.219.138.54:443 with timeout 0

I see that it knows the correct ip address 212.219.138.54 for the the domain www.soas.ac.uk, yet it cannot connect

image
image911×505 16 KB

10

It might be better to upgrade JDK and also Fess. Fess 12 is EOLed.