fess-crawler.log for Postgraduate Module Open Options 2021/22 | SOAS University of London
2021-04-11 23:31:26,381 [WebFsCrawler] INFO Target URL: Postgraduate Module Open Options 2021/22 | SOAS University of London
2021-04-11 23:31:26,381 [WebFsCrawler] INFO Included URL: https://www.soas.ac.uk/students/modulesignup/pg-open-options/.*
2021-04-11 23:31:26,381 [WebFsCrawler] INFO Included URL: (?i).(pdf|PDF)
2021-04-11 23:31:26,382 [WebFsCrawler] INFO Included URL: (?i)..[pP][dD][fF]
2021-04-11 23:31:26,390 [WebFsCrawler] INFO Excluded URL from failures: \Qhttps://www.soas.ac.uk/students/modulesignup/pg-open-options/\E
2021-04-11 23:31:26,391 [WebFsCrawler] DEBUG Crawling Postgraduate Module Open Options 2021/22 | SOAS University of London
2021-04-11 23:31:26,408 [IndexUpdater] DEBUG Starting indexUpdater.
2021-04-11 23:31:26,464 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Queued URL: [UrlQueueImpl [id=rfetgmMBl8hq3_2d5gl8-1.aHR0cHM6Ly93d3cuc29hcy5hYy51ay9zdHVkZW50cy9tb2R1bGVzaWdudXAvcGctb3Blbi1vcHRpb25zLw, sessionId=rfetgmMBl8hq3_2d5gl8-1, method=GET, url=https://www.soas.ac.uk/students/modulesignup/pg-open-options/, encoding=null, parentUrl=null, depth=0, lastModified=0, createTime=1618180286350]]
2021-04-11 23:31:26,491 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG Postgraduate Module Open Options 2021/22 | SOAS University of London is not a target url. (0)
2021-04-11 23:31:36,408 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:31:36,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:31:36,417 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:31:36,417 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:31:36,993 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (1)
2021-04-11 23:31:46,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:31:46,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:31:46,412 [IndexUpdater] INFO Processing no docs (Doc:{access 3ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:31:46,412 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:31:47,495 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (2)
2021-04-11 23:31:56,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:31:56,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:31:56,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:31:56,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:31:57,997 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (3)
2021-04-11 23:32:06,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:06,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:06,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:32:06,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:08,499 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (4)
2021-04-11 23:32:16,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:16,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:16,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 334MB, heap 1GB, max 1GB})
2021-04-11 23:32:16,412 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:19,002 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (5)
2021-04-11 23:32:26,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:26,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:26,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 338MB, heap 1GB, max 1GB})
2021-04-11 23:32:26,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:26,424 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:1383804928,“total”:12430614528},“swap_space”:{“free”:6511763456,“total”:6874460160}},“cpu”:{“percent”:3},“load_averages”:[0.2, 0.11, 0.08]},“process”:{“file_descriptor”:{“open”:292,“max”:65535},“cpu”:{“percent”:0,“total”:22910},“virtual_memory”:{“total”:4093710336}},“jvm”:{“memory”:{“heap”:{“used”:350515296,“committed”:1173094400,“max”:1173094400,“percent”:29},“non_heap”:{“used”:81712320,“committed”:86175744}},“pools”:{“direct”:{“count”:26,“used”:135274497,“capacity”:135274496},“mapped”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:2,“time”:169},“old”:{“count”:1,“time”:76}},“threads”:{“count”:40,“peak”:40},“classes”:{“loaded”:10266,“total_loaded”:10266,“unloaded”:0},“uptime”:75092},“elasticsearch”:{“nodes”:{“TRpYmpykSxSnrDpM3qsPrQ”:{“timestamp”:1618180346189,“name”:“master-data-node-1”,“transport_address”:“127.0.0.1:9300”,“host”:“127.0.0.1”,“ip”:“127.0.0.1:9300”,“roles”:[“master”,“data”,“ingest”],“indices”:{“docs”:{“count”:19143930,“deleted”:6996},“store”:{“size_in_bytes”:4492337313},“indexing”:{“index_total”:1492274,“index_time_in_millis”:1174300,“index_current”:0,“index_failed”:99,“delete_total”:1506785,“delete_time_in_millis”:22959,“delete_current”:0,“noop_update_total”:0,“is_throttled”:false,“throttle_time_in_millis”:0},“get”:{“total”:1397321,“time_in_millis”:49687,“exists_total”:1250571,“exists_time_in_millis”:47131,“missing_total”:146750,“missing_time_in_millis”:2556,“current”:0},“search”:{“open_contexts”:15,“query_total”:4433299,“query_time_in_millis”:523374,“query_current”:0,“fetch_total”:201679,“fetch_time_in_millis”:155520,“fetch_current”:0,“scroll_total”:89888,“scroll_time_in_millis”:8133796822,“scroll_current”:15,“suggest_total”:0,“suggest_time_in_millis”:0,“suggest_current”:0},“merges”:{“current”:0,“current_docs”:0,“current_size_in_bytes”:0,“total”:9644,“total_time_in_millis”:538127,“total_docs”:25105935,“total_size_in_bytes”:12574770804,“total_stopped_time_in_millis”:0,“total_throttled_time_in_millis”:0,“total_auto_throttle_in_bytes”:4131389440},“refresh”:{“total”:663078,“total_time_in_millis”:935154,“listeners”:0},“flush”:{“total”:20387,“total_time_in_millis”:159900},“warmer”:{“current”:0,“total”:118878,“total_time_in_millis”:1553},“query_cache”:{“memory_size_in_bytes”:30950,“total_count”:1405,“hit_count”:1036,“miss_count”:369,“cache_size”:20,“cache_count”:80,“evictions”:60},“fielddata”:{“memory_size_in_bytes”:136144,“evictions”:0},“completion”:{“size_in_bytes”:0},“segments”:{“count”:445,“memory_in_bytes”:11717367,“terms_memory_in_bytes”:9832226,“stored_fields_memory_in_bytes”:1424320,“term_vectors_memory_in_bytes”:11392,“norms_memory_in_bytes”:16704,“points_memory_in_bytes”:235217,“doc_values_memory_in_bytes”:197508,“index_writer_memory_in_bytes”:0,“version_map_memory_in_bytes”:229,“fixed_bit_set_memory_in_bytes”:0,“max_unsafe_auto_id_timestamp”:-1,“file_sizes”:{}},“translog”:{“operations”:4807,“size_in_bytes”:57003890,“uncommitted_operations”:13,“uncommitted_size_in_bytes”:10407},“request_cache”:{“memory_size_in_bytes”:1705056,“evictions”:0,“hit_count”:1581245,“miss_count”:1503030},“recovery”:{“current_as_source”:0,“current_as_target”:0,“throttle_time_in_millis”:0}},“os”:{“timestamp”:1618180346361,“cpu”:{“percent”:0,“load_average”:{“1m”:0.2,“5m”:0.11,“15m”:0.08}},“mem”:{“total_in_bytes”:12430614528,“free_in_bytes”:1382567936,“used_in_bytes”:11048046592,“free_percent”:11,“used_percent”:89},“swap”:{“total_in_bytes”:6874460160,“free_in_bytes”:6511763456,“used_in_bytes”:362696704},“cgroup”:{“cpuacct”:{“control_group”:"/",“usage_nanos”:149380652700038},“cpu”:{“control_group”:"/",“cfs_period_micros”:100000,“cfs_quota_micros”:-1,“stat”:{“number_of_elapsed_periods”:0,“number_of_times_throttled”:0,“time_throttled_nanos”:0}},“memory”:{“control_group”:"/",“limit_in_bytes”:“9223372036854771712”,“usage_in_bytes”:“9981755392”}}},“process”:{“timestamp”:1618180346361,“open_file_descriptors”:1059,“max_file_descriptors”:65536,“cpu”:{“percent”:0,“total_in_millis”:9139140},“mem”:{“total_virtual_in_bytes”:12778676224}},“jvm”:{“timestamp”:1618180346362,“uptime_in_millis”:474020470,“mem”:{“heap_used_in_bytes”:1694947144,“heap_used_percent”:39,“heap_committed_in_bytes”:4260102144,“heap_max_in_bytes”:4260102144,“non_heap_used_in_bytes”:149709040,“non_heap_committed_in_bytes”:157642752,“pools”:{“young”:{“used_in_bytes”:130256160,“max_in_bytes”:279183360,“peak_used_in_bytes”:279183360,“peak_max_in_bytes”:279183360},“survivor”:{“used_in_bytes”:4667656,“max_in_bytes”:34865152,“peak_used_in_bytes”:34865152,“peak_max_in_bytes”:34865152},“old”:{“used_in_bytes”:1560023328,“max_in_bytes”:3946053632,“peak_used_in_bytes”:2973950728,“peak_max_in_bytes”:3946053632}}},“threads”:{“count”:67,“peak_count”:72},“gc”:{“collectors”:{“young”:{“collection_count”:8998,“collection_time_in_millis”:138825},“old”:{“collection_count”:4,“collection_time_in_millis”:285}}},“buffer_pools”:{“direct”:{“count”:107,“used_in_bytes”:139827002,“total_capacity_in_bytes”:139827001},“mapped”:{“count”:780,“used_in_bytes”:4483412444,“total_capacity_in_bytes”:4483412444}},“classes”:{“current_loaded_count”:13070,“total_loaded_count”:13225,“total_unloaded_count”:155}},“thread_pool”:{“bulk”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:117854},“fetch_shard_started”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:8,“completed”:172},“fetch_shard_store”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“flush”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:26326},“force_merge”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“generic”:{“threads”:5,“queue”:0,“active”:0,“rejected”:0,“largest”:5,“completed”:866196},“get”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:1394919},“index”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:52679},“listener”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“management”:{“threads”:4,“queue”:0,“active”:1,“rejected”:0,“largest”:4,“completed”:64848},“refresh”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:23315644},“search”:{“threads”:7,“queue”:0,“active”:0,“rejected”:0,“largest”:7,“completed”:5552954},“snapshot”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“warmer”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0}},“fs”:{“timestamp”:1618180346363,“total”:{“total_in_bytes”:178636283904,“free_in_bytes”:160689635328,“available_in_bytes”:160689635328},“least_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689635328,“used_disk_percent”:10.046474424896019},“most_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689635328,“used_disk_percent”:10.046474424896019},“data”:[{“path”:"/var/lib/elasticsearch/nodes/0",“mount”:"/ (rootfs)",“type”:“rootfs”,“total_in_bytes”:178636283904,“free_in_bytes”:160689635328,“available_in_bytes”:160689635328}],“io_stats”:{“devices”:[{“device_name”:“dm-0”,“operations”:1433778,“read_operations”:136729,“write_operations”:1297049,“read_kilobytes”:4487412,“write_kilobytes”:29663235}],“total”:{“operations”:1433778,“read_operations”:136729,“write_operations”:1297049,“read_kilobytes”:4487412,“write_kilobytes”:29663235}}},“transport”:{“server_open”:39,“rx_count”:2623004,“rx_size_in_bytes”:3215730451,“tx_count”:2623000,“tx_size_in_bytes”:4396458452}}}},“timestamp”:1618180346424}
2021-04-11 23:32:29,504 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (6)
2021-04-11 23:32:36,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:36,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:36,412 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:32:36,412 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:40,006 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (7)
2021-04-11 23:32:46,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:46,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:46,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:32:46,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:32:50,508 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (8)
2021-04-11 23:32:56,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:32:56,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:32:56,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:32:56,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:01,010 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (9)
2021-04-11 23:33:06,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:06,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:06,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:33:06,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:11,512 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (10)
2021-04-11 23:33:16,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:16,409 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:16,411 [IndexUpdater] INFO Processing no docs (Doc:{access 2ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:33:16,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:22,014 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (11)
2021-04-11 23:33:26,409 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:26,410 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:26,411 [IndexUpdater] INFO Processing no docs (Doc:{access 1ms}, Mem:{used 340MB, heap 1GB, max 1GB})
2021-04-11 23:33:26,411 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:26,610 [CoreLib-TimeoutManager] INFO [SYSTEM MONITOR] {“os”:{“memory”:{“physical”:{“free”:1380499456,“total”:12430614528},“swap_space”:{“free”:6511763456,“total”:6874460160}},“cpu”:{“percent”:0},“load_averages”:[0.07, 0.09, 0.07]},“process”:{“file_descriptor”:{“open”:292,“max”:65535},“cpu”:{“percent”:0,“total”:23680},“virtual_memory”:{“total”:4093710336}},“jvm”:{“memory”:{“heap”:{“used”:357766696,“committed”:1173094400,“max”:1173094400,“percent”:30},“non_heap”:{“used”:83091712,“committed”:87224320}},“pools”:{“direct”:{“count”:26,“used”:135274497,“capacity”:135274496},“mapped”:{“count”:0,“used”:0,“capacity”:0}},“gc”:{“young”:{“count”:2,“time”:169},“old”:{“count”:1,“time”:76}},“threads”:{“count”:40,“peak”:40},“classes”:{“loaded”:10425,“total_loaded”:10425,“unloaded”:0},“uptime”:135365},“elasticsearch”:{“nodes”:{“TRpYmpykSxSnrDpM3qsPrQ”:{“timestamp”:1618180406436,“name”:“master-data-node-1”,“transport_address”:“127.0.0.1:9300”,“host”:“127.0.0.1”,“ip”:“127.0.0.1:9300”,“roles”:[“master”,“data”,“ingest”],“indices”:{“docs”:{“count”:19143930,“deleted”:6996},“store”:{“size_in_bytes”:4492337313},“indexing”:{“index_total”:1492274,“index_time_in_millis”:1174300,“index_current”:0,“index_failed”:99,“delete_total”:1506785,“delete_time_in_millis”:22959,“delete_current”:0,“noop_update_total”:0,“is_throttled”:false,“throttle_time_in_millis”:0},“get”:{“total”:1397321,“time_in_millis”:49687,“exists_total”:1250571,“exists_time_in_millis”:47131,“missing_total”:146750,“missing_time_in_millis”:2556,“current”:0},“search”:{“open_contexts”:10,“query_total”:4433390,“query_time_in_millis”:523375,“query_current”:0,“fetch_total”:201685,“fetch_time_in_millis”:155520,“fetch_current”:0,“scroll_total”:89903,“scroll_time_in_millis”:8135287623,“scroll_current”:10,“suggest_total”:0,“suggest_time_in_millis”:0,“suggest_current”:0},“merges”:{“current”:0,“current_docs”:0,“current_size_in_bytes”:0,“total”:9644,“total_time_in_millis”:538127,“total_docs”:25105935,“total_size_in_bytes”:12574770804,“total_stopped_time_in_millis”:0,“total_throttled_time_in_millis”:0,“total_auto_throttle_in_bytes”:4131389440},“refresh”:{“total”:663078,“total_time_in_millis”:935154,“listeners”:0},“flush”:{“total”:20387,“total_time_in_millis”:159900},“warmer”:{“current”:0,“total”:118878,“total_time_in_millis”:1553},“query_cache”:{“memory_size_in_bytes”:30950,“total_count”:1405,“hit_count”:1036,“miss_count”:369,“cache_size”:20,“cache_count”:80,“evictions”:60},“fielddata”:{“memory_size_in_bytes”:136144,“evictions”:0},“completion”:{“size_in_bytes”:0},“segments”:{“count”:445,“memory_in_bytes”:11717367,“terms_memory_in_bytes”:9832226,“stored_fields_memory_in_bytes”:1424320,“term_vectors_memory_in_bytes”:11392,“norms_memory_in_bytes”:16704,“points_memory_in_bytes”:235217,“doc_values_memory_in_bytes”:197508,“index_writer_memory_in_bytes”:0,“version_map_memory_in_bytes”:229,“fixed_bit_set_memory_in_bytes”:0,“max_unsafe_auto_id_timestamp”:-1,“file_sizes”:{}},“translog”:{“operations”:4801,“size_in_bytes”:56999549,“uncommitted_operations”:13,“uncommitted_size_in_bytes”:10407},“request_cache”:{“memory_size_in_bytes”:1705056,“evictions”:0,“hit_count”:1581245,“miss_count”:1503030},“recovery”:{“current_as_source”:0,“current_as_target”:0,“throttle_time_in_millis”:0}},“os”:{“timestamp”:1618180406592,“cpu”:{“percent”:0,“load_average”:{“1m”:0.07,“5m”:0.09,“15m”:0.07}},“mem”:{“total_in_bytes”:12430614528,“free_in_bytes”:1380515840,“used_in_bytes”:11050098688,“free_percent”:11,“used_percent”:89},“swap”:{“total_in_bytes”:6874460160,“free_in_bytes”:6511763456,“used_in_bytes”:362696704},“cgroup”:{“cpuacct”:{“control_group”:"/",“usage_nanos”:149382755682171},“cpu”:{“control_group”:"/",“cfs_period_micros”:100000,“cfs_quota_micros”:-1,“stat”:{“number_of_elapsed_periods”:0,“number_of_times_throttled”:0,“time_throttled_nanos”:0}},“memory”:{“control_group”:"/",“limit_in_bytes”:“9223372036854771712”,“usage_in_bytes”:“9983299584”}}},“process”:{“timestamp”:1618180406592,“open_file_descriptors”:1056,“max_file_descriptors”:65536,“cpu”:{“percent”:0,“total_in_millis”:9139920},“mem”:{“total_virtual_in_bytes”:12778676224}},“jvm”:{“timestamp”:1618180406593,“uptime_in_millis”:474080701,“mem”:{“heap_used_in_bytes”:1724618960,“heap_used_percent”:40,“heap_committed_in_bytes”:4260102144,“heap_max_in_bytes”:4260102144,“non_heap_used_in_bytes”:149709040,“non_heap_committed_in_bytes”:157642752,“pools”:{“young”:{“used_in_bytes”:159927976,“max_in_bytes”:279183360,“peak_used_in_bytes”:279183360,“peak_max_in_bytes”:279183360},“survivor”:{“used_in_bytes”:4667656,“max_in_bytes”:34865152,“peak_used_in_bytes”:34865152,“peak_max_in_bytes”:34865152},“old”:{“used_in_bytes”:1560023328,“max_in_bytes”:3946053632,“peak_used_in_bytes”:2973950728,“peak_max_in_bytes”:3946053632}}},“threads”:{“count”:67,“peak_count”:72},“gc”:{“collectors”:{“young”:{“collection_count”:8998,“collection_time_in_millis”:138825},“old”:{“collection_count”:4,“collection_time_in_millis”:285}}},“buffer_pools”:{“direct”:{“count”:107,“used_in_bytes”:139827002,“total_capacity_in_bytes”:139827001},“mapped”:{“count”:780,“used_in_bytes”:4483412444,“total_capacity_in_bytes”:4483412444}},“classes”:{“current_loaded_count”:13070,“total_loaded_count”:13225,“total_unloaded_count”:155}},“thread_pool”:{“bulk”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:117854},“fetch_shard_started”:{“threads”:1,“queue”:0,“active”:0,“rejected”:0,“largest”:8,“completed”:172},“fetch_shard_store”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“flush”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:26326},“force_merge”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“generic”:{“threads”:5,“queue”:0,“active”:0,“rejected”:0,“largest”:5,“completed”:866329},“get”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:1394919},“index”:{“threads”:4,“queue”:0,“active”:0,“rejected”:0,“largest”:4,“completed”:52679},“listener”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“management”:{“threads”:4,“queue”:0,“active”:1,“rejected”:0,“largest”:4,“completed”:64857},“refresh”:{“threads”:2,“queue”:0,“active”:0,“rejected”:0,“largest”:2,“completed”:23318531},“search”:{“threads”:7,“queue”:0,“active”:0,“rejected”:0,“largest”:7,“completed”:5553074},“snapshot”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0},“warmer”:{“threads”:0,“queue”:0,“active”:0,“rejected”:0,“largest”:0,“completed”:0}},“fs”:{“timestamp”:1618180406594,“total”:{“total_in_bytes”:178636283904,“free_in_bytes”:160689659904,“available_in_bytes”:160689659904},“least_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689659904,“used_disk_percent”:10.046460667332624},“most_usage_estimate”:{“path”:"/var/lib/elasticsearch/nodes/0",“total_in_bytes”:178636283904,“available_in_bytes”:160689659904,“used_disk_percent”:10.046460667332624},“data”:[{“path”:"/var/lib/elasticsearch/nodes/0",“mount”:"/ (rootfs)",“type”:“rootfs”,“total_in_bytes”:178636283904,“free_in_bytes”:160689659904,“available_in_bytes”:160689659904}],“io_stats”:{“devices”:[{“device_name”:“dm-0”,“operations”:1433860,“read_operations”:136729,“write_operations”:1297131,“read_kilobytes”:4487412,“write_kilobytes”:29663744}],“total”:{“operations”:1433860,“read_operations”:136729,“write_operations”:1297131,“read_kilobytes”:4487412,“write_kilobytes”:29663744}}},“transport”:{“server_open”:39,“rx_count”:2623100,“rx_size_in_bytes”:3215739314,“tx_count”:2623096,“tx_size_in_bytes”:4396592047}}}},“timestamp”:1618180406610}
2021-04-11 23:33:32,516 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (12)
2021-04-11 23:33:36,410 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:36,411 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.
2021-04-11 23:33:36,413 [IndexUpdater] INFO Processing no docs (Doc:{access 1ms}, Mem:{used 342MB, heap 1GB, max 1GB})
2021-04-11 23:33:36,413 [IndexUpdater] DEBUG Processed documents in IndexUpdater queue.
2021-04-11 23:33:43,018 [Crawler-rfetgmMBl8hq3_2d5gl8-1-1] DEBUG The url is null. (13)
2021-04-11 23:33:46,411 [IndexUpdater] DEBUG Processing documents in IndexUpdater queue.
2021-04-11 23:33:46,411 [IndexUpdater] DEBUG Getting documents in IndexUpdater queue.