crawl other site other than in included URL

English:Fess
1

(from github.com/RSeptiane)
Hi,

I can do crawling on my website but when I saw the result, there were some other websites have been crawled such as youtube, facebook also SSL website as well and many another outside website.

So my question is how to prevent the crawler to crawl outside website other than my website? if I need to specify in excluded URL, it will be a nightmare because of a lot of them.

Could you please advise how to do it? greatly appreciate it.

Thank you.

2

(from github.com/marevol)
Not reproduced…
Could you provide an example of the crawling configuration?

3

(from github.com/RSeptiane)
Hi,

I’ve set my URL as https://www.mysite.com.au
and there is no excluded URLs.

Here’s what I got in my fees-crawler.log:
2017-12-08 00:00:00,470 [Crawler-20171206153915-1-1] INFO Crawling URL: http://www.facebook.com/sharer.php?u=http://www.nationalrail.co.uk/1924.aspx&t=Help
2017-12-08 00:00:01,436 [Crawler-20171206153915-1-1] INFO Redirect to URL: https://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.nationalrail.co.uk%2F1924.aspx&t=Help
2017-12-08 00:00:11,680 [Crawler-20171206153915-1-1] INFO Crawling URL: http://reddit.com/submit?url=http://www.nationalrail.co.uk/1924.aspx&title=Help
2017-12-08 00:00:11,703 [Crawler-20171206153915-1-1] INFO Redirect to URL: https://www.reddit.com/submit?url=http://www.nationalrail.co.uk/1924.aspx&title=Help
2017-12-08 00:00:12,434 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms, cleanup 44ms}, Mem:{used 185MB, heap 310MB, max 494MB})
2017-12-08 00:00:21,968 [Crawler-20171206153915-1-1] INFO Crawling URL: http://www.stumbleupon.com/submit?url=http://www.nationalrail.co.uk/1924.aspx&Help
2017-12-08 00:00:22,436 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms, cleanup 44ms}, Mem:{used 173MB, heap 310MB, max 494MB})
2017-12-08 00:00:32,436 [IndexUpdater] INFO Processing 1/1 docs (Doc:{access 3ms, cleanup 44ms}, Mem:{used 174MB, heap 310MB, max 494MB})
2017-12-08 00:00:32,488 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms, cleanup 43ms}, Mem:{used 174MB, heap 310MB, max 494MB})
2017-12-08 00:00:32,508 [IndexUpdater] INFO Sent 1 docs (Doc:{process 6ms, send 20ms, size 22KB}, Mem:{used 175MB, heap 310MB, max 494MB})
2017-12-08 00:00:40,289 [Crawler-20171206153915-1-1] INFO Crawling URL: http://del.icio.us/post?url=http://www.nationalrail.co.uk/1924.aspx&Help
2017-12-08 00:00:40,781 [Crawler-20171206153915-1-1] INFO Redirect to URL: https://del.icio.us/post?url=http://www.nationalrail.co.uk/1924.aspx&Help

How I can prevent the crawler to crawl other website than my site? Thank you

4

(from github.com/marevol)
What is the setting of Included URL?

5

(from github.com/RSeptiane)
Hi,

I’ve set empty in the included URL. I thought it will be default as in URL.
Should I set to https://www.mysite.com as well in included URL?

Thank you.

6

(from github.com/RSeptiane)
Hi,

When I set the included URL, the crawling doesn’t work.
Please advise. Thank you.

Here’s the fess.crawler.log.
2017-12-12 12:09:15,975 [WebFsCrawler] INFO Connected to localhost:9300
2017-12-12 12:09:16,085 [WebFsCrawler] INFO Target URL: https://www.mysite.com
2017-12-12 12:09:16,085 [WebFsCrawler] INFO Included URL: https://www.mysite.com/.*
2017-12-12 12:09:26,118 [IndexUpdater] INFO Processing no docs (Doc:{access 8ms}, Mem:{used 148MB, heap 247MB, max 494MB})
2017-12-12 12:09:36,103 [IndexUpdater] INFO Processing no docs (Doc:{access 4ms}, Mem:{used 148MB, heap 247MB, max 494MB})
2017-12-12 12:09:46,101 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 148MB, heap 247MB, max 494MB})
2017-12-12 12:09:56,103 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 148MB, heap 247MB, max 494MB})
2017-12-12 12:10:06,103 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 148MB, heap 247MB, max 494MB})
2017-12-12 12:10:16,103 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 150MB, heap 247MB, max 494MB})
2017-12-12 12:10:16,487 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:3117236224,“total”:17179398144},“swap_space”:{“free”:4012888064,“total”:21540741120}},“cpu”:{“percent”:0},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:22,“total”:32234},“virtual_memory”:{“total”:539176960}},“jvm”:{“memory”:{“heap”:{“used”:156405096,“committed”:259522560,“max”:518979584,“percent”:30},“non_heap”:{“used”:81321920,“committed”:85581824}},“pools”:{“direct”:{“count”:24,“used”:84934689,“capacity”:84934688},“mapped”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:9,“time”:212},“old”:{“count”:2,“time”:66}},“threads”:{“count”:41,“peak”:41},“classes”:{“loaded”:10260,“total_loaded”:10265,“unloaded”:5},“uptime”:73944},“elasticsearch”:{“nodes”:{“KX4EagQ5Q2qs6VQNrEXBwA”:{“timestamp”:1513041016035,“name”:“COLO-WEB-PROD”,“transport_address”:“127.0.0.1:9300”,“host”:“127.0.0.1”,“ip”:“127.0.0.1:9300”,“roles”:[“master”,“data”,“ingest”],“indices”:{“docs”:{“count”:185334,“deleted”:101021},“store”:{“size_in_bytes”:76843897,“throttle_time_in_millis”:0},“indexing”:{“index_total”:2234719,“index_time_in_millis”:3019819,“index_current”:0,“index_failed”:2,“delete_total”:226064,“delete_time_in_millis”:13153,“delete_current”:0,“noop_update_total”:71,“is_throttled”:false,“throttle_time_in_millis”:0},“get”:{“total”:1914158,“time_in_millis”:169755,“exists_total”:1749182,“exists_time_in_millis”:161647,“missing_total”:164976,“missing_time_in_millis”:8108,“current”:0},“search”:{“open_contexts”:75,“query_total”:28785111,“query_time_in_millis”:2633209,“query_current”:0,“fetch_total”:412459,“fetch_time_in_millis”:3022131,“fetch_current”:0,“scroll_total”:94251,“scroll_time_in_millis”:8809326870,“scroll_current”:75,“suggest_total”:0,“suggest_time_in_millis”:0,“suggest_current”:0},“merges”:{“current”:0,“current_docs”:0,“current_size_in_bytes”:0,“total”:28886,“total_time_in_millis”:18504062,“total_docs”:652416641,“total_size_in_bytes”:189475377513,“total_stopped_time_in_millis”:0,“total_throttled_time_in_millis”:21847,“total_auto_throttle_in_bytes”:1382166547},“refresh”:{“total”:369740,“total_time_in_millis”:14804883,“listeners”:0},“flush”:{“total”:2058,“total_time_in_millis”:123629},“warmer”:{“current”:0,“total”:340758,“total_time_in_millis”:16284},“query_cache”:{“memory_size_in_bytes”:71826,“total_count”:9158222,“hit_count”:370148,“miss_count”:8788074,“cache_size”:65,“cache_count”:37112,“evictions”:37047},“fielddata”:{“memory_size_in_bytes”:100324,“evictions”:0},“completion”:{“size_in_bytes”:0},“segments”:{“count”:111,“memory_in_bytes”:682697,“terms_memory_in_bytes”:575037,“stored_fields_memory_in_bytes”:59272,“term_vectors_memory_in_bytes”:0,“norms_memory_in_bytes”:768,“points_memory_in_bytes”:4104,“doc_values_memory_in_bytes”:43516,“index_writer_memory_in_bytes”:0,“version_map_memory_in_bytes”:172,“fixed_bit_set_memory_in_bytes”:0,“max_unsafe_auto_id_timestamp”:1512532186882,“file_sizes”:{}},“translog”:{“operations”:15,“size_in_bytes”:7410},“request_cache”:{“memory_size_in_bytes”:0,“evictions”:2735148,“hit_count”:7148910,“miss_count”:19188860},“recovery”:{“current_as_source”:0,“current_as_target”:0,“throttle_time_in_millis”:0}},“os”:{“timestamp”:1513041016373,“cpu”:{“percent”:15},“mem”:{“total_in_bytes”:17179398144,“free_in_bytes”:3104890880,“used_in_bytes”:14074507264,“free_percent”:18,“used_percent”:82},“swap”:{“total_in_bytes”:21540741120,“free_in_bytes”:4002181120,“used_in_bytes”:17538560000}},“process”:{“timestamp”:1513041016383,“open_file_descriptors”:-1,“max_file_descriptors”:-1,“cpu”:{“percent”:0,“total_in_millis”:51383375},“mem”:{“total_virtual_in_bytes”:1523027968}},“jvm”:{“timestamp”:1513041016383,“uptime_in_millis”:508114047,“mem”:{“heap_used_in_bytes”:548037512,“heap_used_percent”:52,“heap_committed_in_bytes”:1038876672,“heap_max_in_bytes”:1038876672,“non_heap_used_in_bytes”:141546488,“non_heap_committed_in_bytes”:148828160,“pools”:{“young”:{“used_in_bytes”:29101960,“max_in_bytes”:279183360,“peak_used_in_bytes”:279183360,“peak_max_in_bytes”:279183360},“survivor”:{“used_in_bytes”:6736944,“max_in_bytes”:34865152,“peak_used_in_bytes”:34865152,“peak_max_in_bytes”:34865152},“old”:{“used_in_bytes”:512198608,“max_in_bytes”:724828160,“peak_used_in_bytes”:723653336,“peak_max_in_bytes”:724828160}}},“threads”:{“count”:59,“peak_count”:65},“gc”:{“collectors”:{“young”:{“collection_count”:20014,“collection_time_in_millis”:424587},“old”:{“collection_count”:392,“collection_time_in_millis”:112428}}},“buffer_pools”:{“direct”:{“count”:59,“used_in_bytes”:139864005,“total_capacity_in_bytes”:139864004},“mapped”:{“count”:161,“used_in_bytes”:74984899,“total_capacity_in_bytes”:74984899}},“classes”:{“current_loaded_count”:12338,“total_loaded_count”:12397,“total_unloaded_count”:59}},“thread_pool”:{“bulk”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:332583},“fetch_shard_started”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:8,“completed”:53},“fetch_shard_store”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“flush”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:3603},“force_merge”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“generic”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:50866},“get”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:1899819},“index”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:43016},“listener”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“management”:{“threads”:4,“queue”:0,“active”:1,“rejected”:0,“largest”:4,“completed”:77129},“refresh”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:5143401},“search”:{“threads”:7,“queue”:0,“active”:0,“rejected”:0,“largest”:7,“completed”:34960671},“snapshot”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“warmer”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0}},“fs”:{“timestamp”:1513041016384,“total”:{“total_in_bytes”:128479916032,“free_in_bytes”:39658119168,“available_in_bytes”:39658119168},“data”:[{“path”:“C:\Elastic\Elasticsearch\data\nodes\0”,“mount”:"(C:)",“type”:“NTFS”,“total_in_bytes”:128479916032,“free_in_bytes”:39658119168,“available_in_bytes”:39658119168}]},“transport”:{“server_open”:39,“rx_count”:8568845,“rx_size_in_bytes”:9137877126,“tx_count”:8568844,“tx_size_in_bytes”:35563594276}}}},“timestamp”:1513041016487}
2017-12-12 12:10:26,107 [IndexUpdater] INFO Processing no docs (Doc:{access 5ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:10:36,104 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:10:46,105 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:10:56,106 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:11:06,118 [IndexUpdater] INFO Processing no docs (Doc:{access 14ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:11:16,109 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:11:16,785 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:3106709504,“total”:17179398144},“swap_space”:{“free”:4003905536,“total”:21540741120}},“cpu”:{“percent”:23},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:33156},“virtual_memory”:{“total”:533655552}},“jvm”:{“memory”:{“heap”:{“used”:160938072,“committed”:259522560,“max”:518979584,“percent”:31},“non_heap”:{“used”:82496840,“committed”:86630400}},“pools”:{“direct”:{“count”:24,“used”:84934689,“capacity”:84934688},“mapped”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:9,“time”:212},“old”:{“count”:2,“time”:66}},“threads”:{“count”:41,“peak”:41},“classes”:{“loaded”:10400,“total_loaded”:10405,“unloaded”:5},“uptime”:134553},“elasticsearch”:{“nodes”:{“KX4EagQ5Q2qs6VQNrEXBwA”:{“timestamp”:1513041076611,“name”:“COLO-WEB-PROD”,“transport_address”:“127.0.0.1:9300”,“host”:“127.0.0.1”,“ip”:“127.0.0.1:9300”,“roles”:[“master”,“data”,“ingest”],“indices”:{“docs”:{“count”:185334,“deleted”:101021},“store”:{“size_in_bytes”:76843897,“throttle_time_in_millis”:0},“indexing”:{“index_total”:2234719,“index_time_in_millis”:3019819,“index_current”:0,“index_failed”:2,“delete_total”:226064,“delete_time_in_millis”:13153,“delete_current”:0,“noop_update_total”:71,“is_throttled”:false,“throttle_time_in_millis”:0},“get”:{“total”:1914158,“time_in_millis”:169755,“exists_total”:1749182,“exists_time_in_millis”:161647,“missing_total”:164976,“missing_time_in_millis”:8108,“current”:0},“search”:{“open_contexts”:20,“query_total”:28785208,“query_time_in_millis”:2633219,“query_current”:0,“fetch_total”:412468,“fetch_time_in_millis”:3022131,“fetch_current”:0,“scroll_total”:94316,“scroll_time_in_millis”:8841994640,“scroll_current”:20,“suggest_total”:0,“suggest_time_in_millis”:0,“suggest_current”:0},“merges”:{“current”:0,“current_docs”:0,“current_size_in_bytes”:0,“total”:28886,“total_time_in_millis”:18504062,“total_docs”:652416641,“total_size_in_bytes”:189475377513,“total_stopped_time_in_millis”:0,“total_throttled_time_in_millis”:21847,“total_auto_throttle_in_bytes”:1382166547},“refresh”:{“total”:369740,“total_time_in_millis”:14804883,“listeners”:0},“flush”:{“total”:2058,“total_time_in_millis”:123629},“warmer”:{“current”:0,“total”:340758,“total_time_in_millis”:16284},“query_cache”:{“memory_size_in_bytes”:71826,“total_count”:9158282,“hit_count”:370208,“miss_count”:8788074,“cache_size”:65,“cache_count”:37112,“evictions”:37047},“fielddata”:{“memory_size_in_bytes”:197156,“evictions”:0},“completion”:{“size_in_bytes”:0},“segments”:{“count”:111,“memory_in_bytes”:682697,“terms_memory_in_bytes”:575037,“stored_fields_memory_in_bytes”:59272,“term_vectors_memory_in_bytes”:0,“norms_memory_in_bytes”:768,“points_memory_in_bytes”:4104,“doc_values_memory_in_bytes”:43516,“index_writer_memory_in_bytes”:0,“version_map_memory_in_bytes”:172,“fixed_bit_set_memory_in_bytes”:0,“max_unsafe_auto_id_timestamp”:1512532186882,“file_sizes”:{}},“translog”:{“operations”:15,“size_in_bytes”:7410},“request_cache”:{“memory_size_in_bytes”:0,“evictions”:2735148,“hit_count”:7148910,“miss_count”:19188860},“recovery”:{“current_as_source”:0,“current_as_target”:0,“throttle_time_in_millis”:0}},“os”:{“timestamp”:1513041076770,“cpu”:{“percent”:23},“mem”:{“total_in_bytes”:17179398144,“free_in_bytes”:3105521664,“used_in_bytes”:14073876480,“free_percent”:18,“used_percent”:82},“swap”:{“total_in_bytes”:21540741120,“free_in_bytes”:4004081664,“used_in_bytes”:17536659456}},“process”:{“timestamp”:1513041076778,“open_file_descriptors”:-1,“max_file_descriptors”:-1,“cpu”:{“percent”:0,“total_in_millis”:51383968},“mem”:{“total_virtual_in_bytes”:1522946048}},“jvm”:{“timestamp”:1513041076778,“uptime_in_millis”:508174442,“mem”:{“heap_used_in_bytes”:558154952,“heap_used_percent”:53,“heap_committed_in_bytes”:1038876672,“heap_max_in_bytes”:1038876672,“non_heap_used_in_bytes”:141546488,“non_heap_committed_in_bytes”:148828160,“pools”:{“young”:{“used_in_bytes”:39219400,“max_in_bytes”:279183360,“peak_used_in_bytes”:279183360,“peak_max_in_bytes”:279183360},“survivor”:{“used_in_bytes”:6736944,“max_in_bytes”:34865152,“peak_used_in_bytes”:34865152,“peak_max_in_bytes”:34865152},“old”:{“used_in_bytes”:512198608,“max_in_bytes”:724828160,“peak_used_in_bytes”:723653336,“peak_max_in_bytes”:724828160}}},“threads”:{“count”:58,“peak_count”:65},“gc”:{“collectors”:{“young”:{“collection_count”:20014,“collection_time_in_millis”:424587},“old”:{“collection_count”:392,“collection_time_in_millis”:112428}}},“buffer_pools”:{“direct”:{“count”:59,“used_in_bytes”:139864005,“total_capacity_in_bytes”:139864004},“mapped”:{“count”:159,“used_in_bytes”:74978001,“total_capacity_in_bytes”:74978001}},“classes”:{“current_loaded_count”:12338,“total_loaded_count”:12397,“total_unloaded_count”:59}},“thread_pool”:{“bulk”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:332583},“fetch_shard_started”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:8,“completed”:53},“fetch_shard_store”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“flush”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:3603},“force_merge”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“generic”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:50872},“get”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:1899819},“index”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:43016},“listener”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“management”:{“threads”:4,“queue”:0,“active”:1,“rejected”:0,“largest”:4,“completed”:77138},“refresh”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:5144017},“search”:{“threads”:7,“queue”:0,“active”:0,“rejected”:0,“largest”:7,“completed”:34960797},“snapshot”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“warmer”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0}},“fs”:{“timestamp”:1513041076778,“total”:{“total_in_bytes”:128479916032,“free_in_bytes”:39656202240,“available_in_bytes”:39656202240},“data”:[{“path”:“C:\Elastic\Elasticsearch\data\nodes\0”,“mount”:"(C:)",“type”:“NTFS”,“total_in_bytes”:128479916032,“free_in_bytes”:39656202240,“available_in_bytes”:39656202240}]},“transport”:{“server_open”:39,“rx_count”:8568943,“rx_size_in_bytes”:9137885941,“tx_count”:8568942,“tx_size_in_bytes”:35563641786}}}},“timestamp”:1513041076785}
2017-12-12 12:11:26,111 [IndexUpdater] INFO Processing no docs (Doc:{access 6ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:11:36,112 [IndexUpdater] INFO Processing no docs (Doc:{access 5ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:11:46,109 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 153MB, heap 247MB, max 494MB})
2017-12-12 12:11:56,110 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 154MB, heap 247MB, max 494MB})
2017-12-12 12:12:06,110 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 154MB, heap 247MB, max 494MB})
2017-12-12 12:12:16,110 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 154MB, heap 247MB, max 494MB})
2017-12-12 12:12:17,248 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:3097755648,“total”:17179398144},“swap_space”:{“free”:4000120832,“total”:21540741120}},“cpu”:{“percent”:22},“load_averages”:null},“process”:{“file_descriptor”:{“open”:-1,“max”:-1},“cpu”:{“percent”:0,“total”:33750},“virtual_memory”:{“total”:534355968}},“jvm”:{“memory”:{“heap”:{“used”:161751344,“committed”:259522560,“max”:518979584,“percent”:31},“non_heap”:{“used”:82906760,“committed”:86892544}},“pools”:{“direct”:{“count”:24,“used”:84934689,“capacity”:84934688},“mapped”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:9,“time”:212},“old”:{“count”:2,“time”:66}},“threads”:{“count”:41,“peak”:41},“classes”:{“loaded”:10400,“total_loaded”:10405,“unloaded”:5},“uptime”:194957},“elasticsearch”:{“nodes”:{“KX4EagQ5Q2qs6VQNrEXBwA”:{“timestamp”:1513041137014,“name”:“COLO-WEB-PROD”,“transport_address”:“127.0.0.1:9300”,“host”:“127.0.0.1”,“ip”:“127.0.0.1:9300”,“roles”:[“master”,“data”,“ingest”],“indices”:{“docs”:{“count”:185335,“deleted”:101021},“store”:{“size_in_bytes”:76856235,“throttle_time_in_millis”:0},“indexing”:{“index_total”:2234721,“index_time_in_millis”:3019825,“index_current”:0,“index_failed”:2,“delete_total”:226064,“delete_time_in_millis”:13153,“delete_current”:0,“noop_update_total”:71,“is_throttled”:false,“throttle_time_in_millis”:0},“get”:{“total”:1914158,“time_in_millis”:169755,“exists_total”:1749182,“exists_time_in_millis”:161647,“missing_total”:164976,“missing_time_in_millis”:8108,“current”:0},“search”:{“open_contexts”:20,“query_total”:28785311,“query_time_in_millis”:2633229,“query_current”:0,“fetch_total”:412477,“fetch_time_in_millis”:3022132,“fetch_current”:0,“scroll_total”:94326,“scroll_time_in_millis”:8843110493,“scroll_current”:20,“suggest_total”:0,“suggest_time_in_millis”:0,“suggest_current”:0},“merges”:{“current”:0,“current_docs”:0,“current_size_in_bytes”:0,“total”:28886,“total_time_in_millis”:18504062,“total_docs”:652416641,“total_size_in_bytes”:189475377513,“total_stopped_time_in_millis”:0,“total_throttled_time_in_millis”:21847,“total_auto_throttle_in_bytes”:1382166547},“refresh”:{“total”:369799,“total_time_in_millis”:14804934,“listeners”:0},“flush”:{“total”:2058,“total_time_in_millis”:123629},“warmer”:{“current”:0,“total”:340760,“total_time_in_millis”:16284},“query_cache”:{“memory_size_in_bytes”:71826,“total_count”:9158337,“hit_count”:370262,“miss_count”:8788075,“cache_size”:65,“cache_count”:37112,“evictions”:37047},“fielddata”:{“memory_size_in_bytes”:197156,“evictions”:0},“completion”:{“size_in_bytes”:0},“segments”:{“count”:112,“memory_in_bytes”:686854,“terms_memory_in_bytes”:578716,“stored_fields_memory_in_bytes”:59584,“term_vectors_memory_in_bytes”:0,“norms_memory_in_bytes”:768,“points_memory_in_bytes”:4114,“doc_values_memory_in_bytes”:43672,“index_writer_memory_in_bytes”:0,“version_map_memory_in_bytes”:0,“fixed_bit_set_memory_in_bytes”:0,“max_unsafe_auto_id_timestamp”:1512532186882,“file_sizes”:{}},“translog”:{“operations”:17,“size_in_bytes”:8333},“request_cache”:{“memory_size_in_bytes”:0,“evictions”:2735148,“hit_count”:7148910,“miss_count”:19188860},“recovery”:{“current_as_source”:0,“current_as_target”:0,“throttle_time_in_millis”:0}},“os”:{“timestamp”:1513041137230,“cpu”:{“percent”:22},“mem”:{“total_in_bytes”:17179398144,“free_in_bytes”:3097763840,“used_in_bytes”:14081634304,“free_percent”:18,“used_percent”:82},“swap”:{“total_in_bytes”:21540741120,“free_in_bytes”:4000104448,“used_in_bytes”:17540636672}},“process”:{“timestamp”:1513041137240,“open_file_descriptors”:-1,“max_file_descriptors”:-1,“cpu”:{“percent”:0,“total_in_millis”:51384609},“mem”:{“total_virtual_in_bytes”:1522946048}},“jvm”:{“timestamp”:1513041137240,“uptime_in_millis”:508234904,“mem”:{“heap_used_in_bytes”:572758640,“heap_used_percent”:55,“heap_committed_in_bytes”:1038876672,“heap_max_in_bytes”:1038876672,“non_heap_used_in_bytes”:141546488,“non_heap_committed_in_bytes”:148828160,“pools”:{“young”:{“used_in_bytes”:53823088,“max_in_bytes”:279183360,“peak_used_in_bytes”:279183360,“peak_max_in_bytes”:279183360},“survivor”:{“used_in_bytes”:6736944,“max_in_bytes”:34865152,“peak_used_in_bytes”:34865152,“peak_max_in_bytes”:34865152},“old”:{“used_in_bytes”:512198608,“max_in_bytes”:724828160,“peak_used_in_bytes”:723653336,“peak_max_in_bytes”:724828160}}},“threads”:{“count”:58,“peak_count”:65},“gc”:{“collectors”:{“young”:{“collection_count”:20014,“collection_time_in_millis”:424587},“old”:{“collection_count”:392,“collection_time_in_millis”:112428}}},“buffer_pools”:{“direct”:{“count”:59,“used_in_bytes”:139864005,“total_capacity_in_bytes”:139864004},“mapped”:{“count”:160,“used_in_bytes”:74986592,“total_capacity_in_bytes”:74986592}},“classes”:{“current_loaded_count”:12338,“total_loaded_count”:12397,“total_unloaded_count”:59}},“thread_pool”:{“bulk”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:332585},“fetch_shard_started”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:8,“completed”:53},“fetch_shard_store”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“flush”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:3603},“force_merge”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“generic”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:50878},“get”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:1899819},“index”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:43016},“listener”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“management”:{“threads”:4,“queue”:0,“active”:1,“rejected”:0,“largest”:4,“completed”:77147},“refresh”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:5144680},“search”:{“threads”:7,“queue”:0,“active”:0,“rejected”:0,“largest”:7,“completed”:34960934},“snapshot”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“warmer”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0}},“fs”:{“timestamp”:1513041137241,“total”:{“total_in_bytes”:128479916032,“free_in_bytes”:39651168256,“available_in_bytes”:39651168256},“data”:[{“path”:“C:\Elastic\Elasticsearch\data\nodes\0”,“mount”:"(C:)",“type”:“NTFS”,“total_in_bytes”:128479916032,“free_in_bytes”:39651168256,“available_in_bytes”:39651168256}]},“transport”:{“server_open”:39,“rx_count”:8569044,“rx_size_in_bytes”:9137896098,“tx_count”:8569043,“tx_size_in_bytes”:35563710918}}}},“timestamp”:1513041137248}
2017-12-12 12:12:26,112 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 154MB, heap 247MB, max 494MB})
2017-12-12 12:12:36,112 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 154MB, heap 247MB, max 494MB})
2017-12-12 12:12:46,137 [IndexUpdater] INFO Processing no docs (Doc:{access 25ms}, Mem:{used 154MB, heap 247MB, max 494MB})
2017-12-12 12:12:46,566 [WebFsCrawler] INFO [EXEC TIME] crawling time: 210673ms
2017-12-12 12:12:56,113 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 155MB, heap 247MB, max 494MB})
2017-12-12 12:12:56,114 [IndexUpdater] INFO [EXEC TIME] index update time: 127ms
2017-12-12 12:12:56,181 [main] INFO Finished Crawler
2017-12-12 12:12:56,315 [main] INFO [CRAWL INFO] CrawlerEndTime=2017-12-12T12:12:56.181+1100,WebFsCrawlExecTime=210673,CrawlerStatus=true,CrawlerStartTime=2017-12-12T12:09:15.813+1100,WebFsCrawlEndTime=2017-12-12T12:12:56.180+1100,WebFsIndexExecTime=127,WebFsIndexSize=0,CrawlerExecTime=220368,WebFsCrawlStartTime=2017-12-12T12:09:15.850+1100
2017-12-12 12:13:01,327 [main] INFO Disconnected to elasticsearch:localhost:9300
2017-12-12 12:13:04,266 [main] INFO Destroyed LaContainer.

7

(from github.com/marevol)
https://www.mysite.com/.* does not match https://www.mysite.com
The proper setting is as below:

URL: https://www.mysite.com/
Included URL: https://www.mysite.com/.*