mysql BLOB type

rafael · August 18, 2023, 1:18pm

Fess crallwer into mysql BLOB type?

url="http:/xxxx.com/view.php?id=" + bug_id
host="http://xxxx.com"
site="http://xxxx.com"
title=filename
filename=filename
content=content
digest=content
mimetype=file_type
anchor=
content_length=content.length()
last_modified=new java.util.Date()

its not working.

Im getting alot of errors like below:

 boost=1.0E10, mimetype=application/datastore, virtual_host=[]}}, crawlingInfoId=20230818000000-1, id=22069, crawler.stats.key=org.codelibs.fess.helper.CrawlerStatsHelper$StatsKeyObject@3e3007ba, username=admin}
groovy.lang.MissingPropertyException: No such property: content for class: Script1
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:67) ~[fess-script-groovy-14.10.0.jar:?]
	at org.codehaus.groovy.vmplugin.v8.IndyGuardsFiltersAndSignatures.unwrap(IndyGuardsFiltersAndSignatures.java:161) ~[fess-script-groovy-14.10.0.jar:?]
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321) ~[fess-script-groovy-14.10.0.jar:?]
	at Script1.run(Script1.groovy:1) ~[?:?]
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:460) ~[fess-script-groovy-14.10.0.jar:?]
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:495) ~[fess-script-groovy-14.10.0.jar:?]
	at groovy.lang.GroovyShell.evaluate(GroovyShell.java:469) ~[fess-script-groovy-14.10.0.jar:?]

shinsuke · August 19, 2023, 5:37am

The current release does not support it.
The snapshot release might work.

rafael · August 20, 2023, 12:27am

With the official fess-ds-db, in crawler log, it shows part of the document, but didnt finished or indexed. With snapshot I get heap memory errors. Should I have memory to download all the blob space at once? If I have 1000 documents, with 30GB total. Shoud I have a heap memory of 30GB?
thank you for your help.
part of snapshot crawler log:

{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":1759}},"fs":{"timestamp":1692414735293,"total":{"total_in_bytes":983425871872,"free_in_bytes":565438902272,"available_in_bytes":565438902272,"cache_reserved_in_bytes":0},"data":[{"mount":"Dados (D:)","total_in_bytes":983425871872,"free_in_bytes":565438902272,"available_in_bytes":565438902272,"cache_utilized":0}]},"transport":{"server_open":0,"total_outbound_connections":0,"rx_count":0,"rx_size_in_bytes":0,"tx_count":0,"tx_size_in_bytes":0},"http":{"current_open":1,"total_opened":13020},"breakers":{"request":{"limit_size_in_bytes":644245094,"limit_size":"614.3mb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":429496729,"limit_size":"409.5mb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.03,"tripped":0},"in_flight_requests":{"limit_size_in_bytes":1073741824,"limit_size":"1gb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":2.0,"tripped":0},"parent":{"limit_size_in_bytes":1073741824,"limit_size":"1gb","estimated_size_in_bytes":652033088,"estimated_size":"621.8mb","overhead":1.0,"tripped":0}},"script":{"compilations":0,"cache_evictions":0,"compilation_limit_triggered":0},"discovery":{"cluster_state_queue":{"total":0,"pending":0,"committed":0},"published_cluster_states":{"full_states":2,"incompatible_diffs":0,"compatible_diffs":46}},"ingest":{"total":{"count":0,"time_in_millis":0,"current":0,"failed":0},"pipelines":{}},"adaptive_selection":{},"script_cache":{"sum":{"compilations":0,"cache_evictions":0,"compilation_limit_triggered":0},"contexts":[]},"indexing_pressure":{"memory":{"current":{"combined_coordinating_and_primary_in_bytes":0,"coordinating_in_bytes":0,"primary_in_bytes":0,"replica_in_bytes":0,"all_in_bytes":0},"total":{"combined_coordinating_and_primary_in_bytes":0,"coordinating_in_bytes":0,"primary_in_bytes":0,"replica_in_bytes":0,"all_in_bytes":0,"coordinating_rejections":0,"primary_rejections":0,"replica_rejections":0},"limit_in_bytes":0}},"shard_indexing_pressure":{"stats":{},"total_rejections_breakup_shadow_mode":{"node_limits":0,"no_successful_request_limits":0,"throughput_degradation_limits":0},"enabled":false,"enforced":false},"search_backpressure":{"search_task":{"resource_tracker_stats":{},"cancellation_stats":{"cancellation_count":0,"cancellation_limit_reached_count":0}},"search_shard_task":{"resource_tracker_stats":{},"cancellation_stats":{"cancellation_count":0,"cancellation_limit_reached_count":0}},"mode":"disabled"}},"cancellation_stats":{"timestamp":0,"name":"","transport_address":"0.0.0.0:0","host":"0.0.0.0","ip":"0.0.0.0:0","roles":[]}}},"timestamp":"2023-08-19T03:12:15.293Z"}
2023-08-19 00:12:44,548 [20230819000000-1] ERROR Failed to process a data crawling: sged
org.codelibs.fess.exception.DataStoreException: Failed to crawl data in DB.
	at org.codelibs.fess.ds.db.DatabaseDataStore.storeData(DatabaseDataStore.java:200) ~[fess-ds-db-14.10.0.jar:?]
	at org.codelibs.fess.ds.AbstractDataStore.store(AbstractDataStore.java:122) ~[classes/:?]
	at org.codelibs.fess.helper.DataIndexHelper$DataCrawlingThread.process(DataIndexHelper.java:218) ~[classes/:?]
	at org.codelibs.fess.helper.DataIndexHelper$DataCrawlingThread.run(DataIndexHelper.java:204) ~[classes/:?]
Caused by: java.sql.SQLException: Java heap space
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:130) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1200) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at org.codelibs.fess.ds.db.DatabaseDataStore.storeData(DatabaseDataStore.java:118) ~[fess-ds-db-14.10.0.jar:?]
	... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.mysql.cj.protocol.a.NativePacketPayload.ensureCapacity(NativePacketPayload.java:135) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativePacketPayload.writeBytes(NativePacketPayload.java:493) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.MultiPacketReader.readMessage(MultiPacketReader.java:91) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.MultiPacketReader.readMessage(MultiPacketReader.java:44) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.ResultsetRowReader.read(ResultsetRowReader.java:75) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.ResultsetRowReader.read(ResultsetRowReader.java:42) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.read(NativeProtocol.java:1648) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:87) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:48) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.read(NativeProtocol.java:1661) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.readAllResults(NativeProtocol.java:1715) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1065) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:998) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:655) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1168) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at org.codelibs.fess.ds.db.DatabaseDataStore.storeData(DatabaseDataStore.java:118) ~[fess-ds-db-14.10.0.jar:?]
	... 3 more
2023-08-19 00:12:44,789 [20230819000000-1] INFO  Deleted 0 old docs.
2023-08-19 00:12:44,789 [DataStoreCrawler] INFO  [EXEC TIME] crawling time: 756781ms
2023-08-19 00:12:44,789 [main] INFO  Finished Crawler
2023-08-19 00:12:45,008 [main] INFO  [CRAWL INFO] DataCrawlExecTime=756781,DataCrawlEndTime=2023-08-19T00:12:44.789-0300,CrawlerEndTime=2023-08-19T00:12:44.789-0300,DataIndexExecTime=0,CrawlerStatus=true,CrawlerStartTime=2023-08-19T00:00:07.827-0300,WebFsCrawlEndTime=2023-08-19T00:00:08.076-0300,DataIndexSize=0,CrawlerExecTime=756962,DataCrawlStartTime=2023-08-19T00:00:07.950-0300,WebFsCrawlStartTime=2023-08-19T00:00:07.950-0300
2023-08-19 00:12:45,024 [main] INFO  Destroyed LaContainer.
2023-08-19 21:07:37,377 [main] INFO  ...Loading specified properties and get by the key: fess_env_crawler.properties, lasta_di.smart.deploy.mode
2023-08-19 21:07:37,429 [main] INFO  ...Setting smart deploy mode: warm
2023-08-19 21:07:37,448 [main] INFO  ...Reading lasta_di.xml
2023-08-19 21:07:37,474 [main] INFO  ...Reading   redefiner.xml
2023-08-19 21:07:37,520 [main] INFO  ...Reading   smartdeploy.xml
2023-08-19 21:07:37,527 [main] INFO  ...Reading     smart/warmdeploy.xml
2023-08-19 21:07:37,530 [main] INFO  ...Reading       convention.xml
2023-08-19 21:07:37,534 [main] INFO  ...Reading         embedded_convention.xml
2023-08-19 21:07:37,546 [main] INFO  ...Reading       creator.xml
2023-08-19 21:07:37,549 [main] INFO  ...Reading         convention.xml (recycle)
2023-08-19 21:07:37,550 [main] INFO  ...Reading         customizer.xml
2023-08-19 21:07:37,553 [main] INFO  ...Reading           lastafw_customizer.xml
2023-08-19 21:07:37,564 [main] INFO  ...Reading           embedded_customizer.xml
2023-08-19 21:07:37,567 [main] INFO  ...Reading             tx_customizer.xml
2023-08-19 21:07:37,577 [main] INFO  ...Reading         my_creator.xml
2023-08-19 21:07:37,581 [main] INFO  ...Reading         lastafw_creator.xml
2023-08-19 21:07:37,585 [main] INFO  ...Reading           convention.xml (recycle)
2023-08-19 21:07:37,585 [main] INFO  ...Reading           customizer.xml (recycle)
2023-08-19 21:07:37,606 [main] INFO  ...Reading         embedded_creator.xml
2023-08-19 21:07:37,609 [main] INFO  ...Reading           convention.xml (recycle)
2023-08-19 21:07:37,609 [main] INFO  ...Reading           customizer.xml (recycle)
2023-08-19 21:07:37,616 [main] INFO  ...Reading       customizer.xml (recycle)
2023-08-19 21:07:37,656 [main] INFO  ...Reading app.xml
2023-08-19 21:07:37,661 [main] INFO  ...Reading   convention.xml
2023-08-19 21:07:37,667 [main] INFO  ...Reading     embedded_convention.xml
2023-08-19 21:07:37,671 [main] INFO  ...Reading   lastaflute_core.xml
2023-08-19 21:07:37,674 [main] INFO  ...Reading     lastaflute_assist.xml
2023-08-19 21:07:37,678 [main] INFO  ...Reading       lastaflute_director.xml
2023-08-19 21:07:37,832 [main] INFO  ...Reading   fess.xml
2023-08-19 21:07:37,835 [main] INFO  ...Reading     fess_config.xml
2023-08-19 21:07:37,841 [main] INFO  ...Reading     fess_ds.xml
2023-08-19 21:07:37,852 [main] INFO  ...Reading       jar:file:/D:/fess-14.10.0/app/WEB-INF/plugin/fess-ds-csv-14.10.0.jar!/fess_ds++.xml
2023-08-19 21:07:37,868 [main] INFO  ...Reading       jar:file:/D:/fess-14.10.0/app/WEB-INF/plugin/fess-ds-db-14.10.1-20230819.053341-2.jar!/fess_ds++.xml
2023-08-19 21:07:37,873 [main] INFO  ...Reading       jar:file:/D:/fess-14.10.0/app/WEB-INF/plugin/fess-ds-elasticsearch-14.10.0.jar!/fess_ds++.xml
2023-08-19 21:07:37,881 [main] INFO  ...Reading       jar:file:/D:/fess-14.10.0/app/WEB-INF/plugin/fess-ds-sharepoint-14.10.0.jar!/fess_ds++.xml
2023-08-19 21:07:37,886 [main] INFO  ...Reading       jar:file:/D:/fess-14.10.0/app/WEB-INF/plugin/fess-ds-wikipedia-14.10.0.jar!/fess_ds++.xml
2023-08-19 21:07:37,892 [main] INFO  ...Reading     fess_se.xml
2023-08-19 21:07:37,899 [main] INFO  ...Reading       jar:file:/D:/fess-14.10.0/app/WEB-INF/plugin/fess-script-groovy-14.10.0.jar!/fess_se++.xml
2023-08-19 21:07:37,904 [main] INFO  ...Reading     esflute_config.xml
2023-08-19 21:07:37,907 [main] INFO  ...Reading       esclient.xml
2023-08-19 21:07:38,152 [main] INFO  ...Reading     esflute_user.xml
2023-08-19 21:07:38,155 [main] INFO  ...Reading       esclient.xml (recycle)
2023-08-19 21:07:38,182 [main] INFO  ...Reading     esflute_log.xml
2023-08-19 21:07:38,185 [main] INFO  ...Reading       esclient.xml (recycle)
2023-08-19 21:07:38,860 [main] INFO  ...Reading   crawler_es.xml
2023-08-19 21:07:38,863 [main] INFO  ...Reading     crawler/container.xml
2023-08-19 21:07:38,866 [main] INFO  ...Reading     crawler/client.xml
2023-08-19 21:07:38,868 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:38,869 [main] INFO  ...Reading       crawler/robotstxt.xml
2023-08-19 21:07:38,870 [main] INFO  ...Reading         crawler/container.xml (recycle)
2023-08-19 21:07:38,874 [main] INFO  ...Reading       crawler/contentlength.xml
2023-08-19 21:07:38,877 [main] INFO  ...Reading         crawler/container.xml (recycle)
2023-08-19 21:07:38,879 [main] INFO  ...Reading       crawler/mimetype.xml
2023-08-19 21:07:38,881 [main] INFO  ...Reading         crawler/container.xml (recycle)
2023-08-19 21:07:38,920 [main] INFO  ...Reading       jar:file:/D:/fess-14.10.0/app/WEB-INF/lib/fess-crawler-playwright-14.10.0.jar!/crawler/client++.xml
2023-08-19 21:07:38,921 [main] INFO  ...Reading         crawler/container.xml
2023-08-19 21:07:38,932 [main] INFO  ...Reading     crawler/rule.xml
2023-08-19 21:07:38,935 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:38,935 [main] INFO  ...Reading       crawler/transformer.xml
2023-08-19 21:07:38,937 [main] INFO  ...Reading         crawler/transformer_basic.xml
2023-08-19 21:07:38,939 [main] INFO  ...Reading           crawler/container.xml (recycle)
2023-08-19 21:07:38,967 [main] INFO  ...Reading     crawler/filter.xml
2023-08-19 21:07:38,969 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:38,971 [main] INFO  ...Reading     crawler/interval.xml
2023-08-19 21:07:38,973 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:38,975 [main] INFO  ...Reading     crawler/extractor.xml
2023-08-19 21:07:38,977 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:38,981 [main] INFO  ...Reading       crawler/extractor+tikaExtractor.xml
2023-08-19 21:07:38,983 [main] INFO  ...Reading         crawler/container.xml
2023-08-19 21:07:39,014 [main] INFO  ...Reading     crawler/mimetype.xml (recycle)
2023-08-19 21:07:39,015 [main] INFO  ...Reading     crawler/encoding.xml
2023-08-19 21:07:39,017 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:39,018 [main] INFO  ...Reading     crawler/urlconverter.xml
2023-08-19 21:07:39,020 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:39,021 [main] INFO  ...Reading     crawler/log.xml
2023-08-19 21:07:39,024 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:39,027 [main] INFO  ...Reading     crawler/sitemaps.xml
2023-08-19 21:07:39,029 [main] INFO  ...Reading       crawler/container.xml (recycle)
2023-08-19 21:07:39,032 [main] INFO  ...Reading     crawler/es.xml
2023-08-19 21:07:39,051 [main] INFO  ...Reading     crawler_es+crawlerThread.xml
2023-08-19 21:07:39,068 [main] INFO  ...Reading     crawler_es+crawlerConfig.xml
2023-08-19 21:07:39,072 [main] INFO  ...Reading   fess_thumbnail.xml
2023-08-19 21:07:39,085 [main] INFO  ...Reading   fess_ingest.xml
2023-08-19 21:07:39,405 [main] INFO  [Objective Config]
2023-08-19 21:07:39,405 [main] INFO   fess_config.properties extends [fess_env_crawler.properties]
2023-08-19 21:07:39,405 [main] INFO   checkImplicitOverride=true, propertyCount=533
2023-08-19 21:07:39,441 [main] INFO  [Exception Translator]
2023-08-19 21:07:39,441 [main] INFO   exceptionTranslationProvider: null
2023-08-19 21:07:39,442 [main] INFO  [Async Manager]
2023-08-19 21:07:39,442 [main] INFO   defaultConcurrentAsyncOption: {no option}
2023-08-19 21:07:39,442 [main] INFO   primaryExecutorService: ThreadPoolExecutor@63c12e52
2023-08-19 21:07:39,442 [main] INFO   secondaryExecutorService: ThreadPoolExecutor@21bd20ee
2023-08-19 21:07:39,442 [main] INFO   tertiaryExecutorService: ThreadPoolExecutor@26c47874
2023-08-19 21:07:39,442 [main] INFO  [Primary Cipher]
2023-08-19 21:07:39,442 [main] INFO   invertibleCryptographer: {AES, UTF-8}
2023-08-19 21:07:39,442 [main] INFO   oneWayCryptographer: {SHA-256, UTF-8}
2023-08-19 21:07:39,444 [main] INFO  [Time Manager]
2023-08-19 21:07:39,444 [main] INFO   businessTimeHandler: TypicalBusinessTimeHandler:{TypicalTimeResourceProvider$$Lambda$149/0x0000024b95352730, TypicalTimeResourceProvider$$Lambda$150/0x0000024b95352b48}
2023-08-19 21:07:39,444 [main] INFO   currentIgnoreTransaction: false
2023-08-19 21:07:39,444 [main] INFO   adjustTimeMillis: 0
2023-08-19 21:07:39,444 [main] INFO   adjustAbsoluteMode: false
2023-08-19 21:07:39,493 [main] INFO  [JSON Manager]
2023-08-19 21:07:39,493 [main] INFO   realJsonParser: GsonJsonEngine
2023-08-19 21:07:39,496 [main] INFO   mapping: {CAMEL_TO_LOWER_SNAKE}
2023-08-19 21:07:39,497 [main] INFO  [Postbox]
2023-08-19 21:07:39,497 [main] INFO   postOffice: PostOffice@1645f294
2023-08-19 21:07:39,497 [main] INFO   postalParkingLot: SMailPostalParkingLot:{{category:{main}=motorbike:{session=javax.mail.Session@6325f352}}}@15c4af7a
2023-08-19 21:07:39,497 [main] INFO   postalPersonnel: FessMailDeliveryDepartmentCreator$1:{SMailConventionReceptionist:{[]}@6cbd0674, batch:{[proofreader:{pmcomment}]}}@55d58825
2023-08-19 21:07:41,371 [main] INFO  Lasta Di boot successfully.
2023-08-19 21:07:41,372 [main] INFO    SmartDeploy Mode: Warm Deploy
2023-08-19 21:07:41,372 [main] INFO    Smart Package: org.codelibs.fess.app
2023-08-19 21:07:41,851 [main] INFO  Starting Crawler..
2023-08-19 21:07:42,177 [WebFsCrawler] INFO  No crawling target urls.
2023-08-19 21:08:45,936 [pool-5-thread-2] INFO  [SYSTEM MONITOR] {"os":{"memory":{"physical":{"free":55437385728,"total":68433997824},"swap_space":{"free":59397816320,"total":78634545152}},"cpu":{"percent":0},"load_averages":null},"process":{"file_descriptor":{"open":-1,"max":-1},"cpu":{"percent":0,"total":18171},"virtual_memory":{"total":4677951488}},"jvm":{"memory":{"heap":{"used":1123024896,"committed":4294967296,"max":10737418240,"percent":10},"non_heap":{"used":71828968,"committed":74711040}},"pools":{"mapped":{"count":0,"used":0,"capacity":0},"direct":{"count":2,"used":139264,"capacity":139264},"mapped - 'non-volatile memory'":{"count":0,"used":0,"capacity":0}},"gc":{"young":{"count":3,"time":63},"old":{"count":0,"time":0}},"threads":{"count":24,"peak":24},"classes":{"loaded":9396,"total_loaded":9396,"unloaded":0},"uptime":69226},"search_engine":{"nodes":{"DUXC_GkTT4uwSSQo9Ynt8A":{"timestamp":1692490125671,"name":"Node 1","transport_address":"127.0.0.1:9300","host":"127.0.0.1","ip":"127.0.0.1:9300","roles":[],"os":{"timestamp":1692490125671,"cpu":{"percent":29},"mem":{"total_in_bytes":68433997824,"free_in_bytes":55238590464,"used_in_bytes":13195407360,"free_percent":81,"used_percent":19},"swap":{"total_in_bytes":78634545152,"free_in_bytes":59382702080,"used_in_bytes":19251843072}},"process":{"timestamp":1692490123576,"open_file_descriptors":-1,"max_file_descriptors":-1,"cpu":{"percent":0,"total_in_millis":6232203},"mem":{"total_virtual_in_bytes":1455501312}},"jvm":{"timestamp":1692490125672,"uptime_in_millis":189374692,"mem":{"heap_used_in_bytes":669478512,"heap_used_percent":62,"heap_committed_in_bytes":867172352,"heap_max_in_bytes":1073741824,"non_heap_used_in_bytes":300523328,"non_heap_committed_in_bytes":325255168,"pools":{"young":{"used_in_bytes":357564416,"max_in_bytes":0,"peak_used_in_bytes":639631360,"peak_max_in_bytes":0,"last_gc_stats":{"used_in_bytes":0,"max_in_bytes":0,"usage_percent":-1}},"old":{"used_in_bytes":302476912,"max_in_bytes":1073741824,"peak_used_in_bytes":691897856,"peak_max_in_bytes":1073741824,"last_gc_stats":{"used_in_bytes":253207664,"max_in_bytes":1073741824,"usage_percent":23}},"survivor":{"used_in_bytes":9437184,"max_in_bytes":0,"peak_used_in_bytes":24524352,"peak_max_in_bytes":0,"last_gc_stats":{"used_in_bytes":9437184,"max_in_bytes":0,"usage_percent":-1}}}},"threads":{"count":114,"peak_count":126},"gc":{"collectors":{"young":{"collection_count":1573,"collection_time_in_millis":7426},"old":{"collection_count":1,"collection_time_in_millis":203}}},"buffer_pools":{"mapped":{"count":134,"used_in_bytes":2015337,"total_capacity_in_bytes":2015337},"direct":{"count":91,"used_in_bytes":17456519,"total_capacity_in_bytes":17456518},"mapped - 'non-volatile memory'":{"count":0,"used_in_bytes":0,"total_capacity_in_bytes":0}},"classes":{"current_loaded_count":32989,"total_loaded_count":0,"total_unloaded_count":9446}},"thread_pool":{"analyze":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":1,"completed":22},"fetch_shard_started":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":16,"completed":58},"fetch_shard_store":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"flush":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":112},"force_merge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"generic":{"threads":15,"queue":0,"active":0,"rejected":0,"largest":15,"completed":503830},"get":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":8},"listener":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"management":{"threads":5,"queue":0,"active":2,"rejected":0,"largest":5,"completed":728365},"refresh":{"threads":4,"queue":0,"active":0,"rejected":0,"largest":4,"completed":7001467},"remote_purge":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"remote_refresh":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"search":{"threads":13,"queue":0,"active":0,"rejected":0,"largest":13,"completed":1182624},"search_throttled":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"system_read":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"system_write":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"translog_sync":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"translog_transfer":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"warmer":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"write":{"threads":8,"queue":0,"active":0,"rejected":0,"largest":8,"completed":1771}},"fs":{"timestamp":1692490125672,"total":{"total_in_bytes":983425871872,"free_in_bytes":565438849024,"available_in_bytes":565438849024,"cache_reserved_in_bytes":0},"data":[{"mount":"Dados (D:)","total_in_bytes":983425871872,"free_in_bytes":565438849024,"available_in_bytes":565438849024,"cache_utilized":0}]},"transport":{"server_open":0,"total_outbound_connections":0,"rx_count":0,"rx_size_in_bytes":0,"tx_count":0,"tx_size_in_bytes":0},"http":{"current_open":6,"total_opened":21500},"breakers":{"request":{"limit_size_in_bytes":644245094,"limit_size":"614.3mb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.0,"tripped":0},"fielddata":{"limit_size_in_bytes":429496729,"limit_size":"409.5mb","estimated_size_in_bytes":0,"estimated_size":"0b","overhead":1.03,"tripped":0},"in_flight_requests":{"limit_size_in_bytes":1073741824,"limit_size":"1gb","estimated_size_in

shinsuke · August 20, 2023, 3:27am

Although I’m not sure if this fix avoids the OOM, I added fetch_size as a parameter.

shinsuke · August 21, 2023, 9:27am

You might need to set MIN_VALUE as fetch_size. I added it in this commit.

rafael · August 21, 2023, 2:13pm

I set heap size to 35GB in fess.in.bat and still getting out of memory.

2023-08-21 11:01:08,640 [20230821104820-1] ERROR Failed to process a data crawling: sged
org.codelibs.fess.exception.DataStoreException: Failed to crawl data in DB.
	at org.codelibs.fess.ds.db.DatabaseDataStore.storeData(DatabaseDataStore.java:234) ~[fess-ds-db-14.10.1-20230821.092541-4.jar:?]
	at org.codelibs.fess.ds.AbstractDataStore.store(AbstractDataStore.java:122) ~[classes/:?]
	at org.codelibs.fess.helper.DataIndexHelper$DataCrawlingThread.process(DataIndexHelper.java:218) ~[classes/:?]
	at org.codelibs.fess.helper.DataIndexHelper$DataCrawlingThread.run(DataIndexHelper.java:204) ~[classes/:?]
Caused by: java.sql.SQLException: Java heap space
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:130) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1200) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at org.codelibs.fess.ds.db.DatabaseDataStore.storeData(DatabaseDataStore.java:152) ~[fess-ds-db-14.10.1-20230821.092541-4.jar:?]
	... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.mysql.cj.protocol.a.NativePacketPayload.ensureCapacity(NativePacketPayload.java:135) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativePacketPayload.writeBytes(NativePacketPayload.java:493) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.MultiPacketReader.readMessage(MultiPacketReader.java:91) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.MultiPacketReader.readMessage(MultiPacketReader.java:44) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.ResultsetRowReader.read(ResultsetRowReader.java:75) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.ResultsetRowReader.read(ResultsetRowReader.java:42) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.read(NativeProtocol.java:1648) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:87) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.TextResultsetReader.read(TextResultsetReader.java:48) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.read(NativeProtocol.java:1661) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.readAllResults(NativeProtocol.java:1715) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:1065) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:998) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.NativeSession.execSQL(NativeSession.java:655) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1168) ~[mysql-connector-j-8.1.0.jar:8.1.0]
	at org.codelibs.fess.ds.db.DatabaseDataStore.storeData(DatabaseDataStore.java:152) ~[fess-ds-db-14.10.1-20230821.092541-4.jar:?]
	... 3 more
2023-08-21 11:01:08,745 [20230821104820-1] INFO  Deleted 0 old docs.
2023-08-21 11:01:08,745 [DataStoreCrawler] INFO  [EXEC TIME] crawling time: 763226ms
2023-08-21 11:01:08,746 [main] INFO  Finished Crawler
2023-08-21 11:01:08,897 [main] INFO  [CRAWL INFO] DataCrawlExecTime=763226,DataCrawlEndTime=2023-08-21T11:01:08.746-0300,CrawlerEndTime=2023-08-21T11:01:08.747-0300,DataIndexExecTime=0,CrawlerStatus=true,CrawlerStartTime=2023-08-21T10:48:25.449-0300,WebFsCrawlEndTime=2023-08-21T10:48:25.524-0300,DataIndexSize=0,CrawlerExecTime=763298,DataCrawlStartTime=2023-08-21T10:48:25.501-0300,WebFsCrawlStartTime=2023-08-21T10:48:25.500-0300
2023-08-21 11:01:08,911 [main] INFO  Destroyed LaContainer.

shinsuke · August 21, 2023, 9:15pm

The setting is fetch_size=MIN_VALUE.

rafael · August 21, 2023, 10:31pm

Yes, with this setting its working now. But indexing is getting weird characters, do you have any ideia why?
For example this pdf is getting wrong indexing, but if I export it to an folder and then index, it is right.

indexing from blob

indexing from a folder

shinsuke · August 22, 2023, 11:53am

I see… extractorFactory should be used. I fixed it in this fix.

rafael · August 22, 2023, 1:25pm

Hi, its still not showing content, should I set something ?

and, is there a way to limit digest? its copying all content to digest, dont know if its right.

Thank you for your help.

shinsuke · August 23, 2023, 11:47am

It seems not to detect a file type from the binary data.
You need to check a debug log for this log message.

rafael · August 23, 2023, 5:05pm

part of log:

"anchor":"","boost":"1.0E32","lang":"en","click_count":0,"config_id":"D74FeA4oBxGcNHEFiSVM8","content":"%PDF-1.4\n2 0 obj\n<< \n/F00 4 0 R\n/F10 9 0 R\n/F20 14 0 R\n/F30 19 0 R\n/F40 24 0 R\n/F50 29 0 R\n/F60 34 0 R\n/F70 39 0 R\n>>\nendobj\n3 0 obj\n<< \n/Type /FontDescriptor\n/Ascent 891\n/CapHeight 0\n/Descent 216\n/Flags 32\n/ItalicAngle 0\n/StemV 0\n/FontBBox [-558 -307 2000 1026]\n/FontName /TimesNewRomanPS-BoldMT\n>>\nendobj\n4 0 obj\n<< \n/Type /Font\n/Subtype /TrueType\n/FirstChar 32\n/LastChar 55\n/Encoding 5 0 R\n/Widths 6 0 R\n/ToUnicode 7 0 R\n/FontDescriptor 3 0 R\n/BaseFont /TimesNewRomanPS-BoldMT\n>>\nendobj\n5 0 obj\n<< \n/Type /Encoding\n/BaseEncoding /WinAnsiEncoding\n/Differences [32 /quotedbl /semicolon /A /B /C /D /E /G /I /L /M /N /O /P /R /S /T /U /e /i /n /t /Eacute /Uacute]\n>>\nendobj\n6 0 obj\n[555 333 722 667 722 722 667 778 389 667 944 722 778 611 722 556 667 722 444 278 556 333 667 722]\nendobj\n7 0 obj\n<< \n/Filter /FlateDecode\n/Length 358\n>>\nstream\nx�U��n�0\u0010\u0005�=_�e��1<\u0012\t�IYdч��\u0001`�#�b�!��}��4R7�|�������ō���O?�7eGg<���kV\u0003_G��L�Qo�]X��/{�^��׍������:��������ж/��\u000Fo؏�z�>w�w�e��ݦ\u0012\u00155�2l����彟X�m\u001B$}�g6�.�f߻+�H�F��6*bg���\\�\u0006+�:��$I�5;��4\u0000\r�\f�\u0005�S\u0000\u0001H ��\\�\u0000\u0005�\u0010�\u0001%�\u0014(\u0000\u0015�\u0012�\u0000G�Q�\u00048\u0001N\u0002\u001A�\u0003z\u0001\u0003\u0018\u0000�\u0000\u00034@\u000BX�\u0001�\u0000E\u0002`\u0000\u000B��-�\n�uB\u0014$y\u0014h�\u0010\u0005I\u001E\u0005Z'DA�G\u0019\u0000Q��Q�uB\u0014$y�h�\u0010\u0005I\u001EU�\u0014Q��C\t� ���\r^����a��\u0003�o���\u0011\u0006.�\u0005�at���e^��%�\u0005\u001D\u0016�C\nendstream\r\n8 0 obj\n<< \n/Type /FontDescriptor\n/Ascent 891\n/CapHeight 0\n/Descent 216\n/Flags 32\n/ItalicAngle 0\n/StemV 0\n/FontBBox [-568 -307 2000 1007]\n/FontName /TimesNewRomanPSMT\n>>\nendobj\n9 0 obj\n<< \n/Type /Font\n/Subtype /TrueType\n/FirstChar 32\n/LastChar 78\n/Encoding 10 0 R\n/Widths 11 0 R\n/ToUnicode 12 0 R\n/FontDescriptor 8 0 R\n/BaseFont /TimesNewRomanPSMT\n>>\nendobj\n10 0 obj\n<< \n/Type /Encoding\n/BaseEncoding /WinAnsiEncoding\n/Differences [32 /parenleft /parenright /hyphen /period /zero /one /two /three /four /five /seven /eight /nine /colon /A /C /D /E /F /G /I /J /L /M /N /O /P /R /S /T /U /V /X /a /d /e /g /l /n /o /r /t /u /degree /Atilde /Ccedilla /agrave]\n>>\nendobj\n11 0 obj\n[333 333 333 250 500 500 500 500 500 500 500 500 500 278 722 667 722 611 556 722 333 389 611 889 722 722 556 667 556 611 722 722 722 444 500 444 500 278 500 500 333 278 500 400 722 667 444]\nendobj\n12 0 obj\n<< \n/Filter /FlateDecode\n/Length 504\n>>\nstream\nx�U�͎�@\u0014��=WQ˞��9��\u00187=.\\�O��\u0005 \u0014�dD���\u001F��z:��\t�\u0002���y;}=��\u0012����ޞ�\u0012�a����?�6�K�\u000Ec�\u0016�\u001B���*}��fZoZo?�?�x;��=d�}���~�X���r<~Y�c��<�ח�og���i�\u0013oq\\B\u001E��!t�_\u001F����7�\u00186�c*ۏ�ܻ���6��x��\u001B��\u0010�}\bY\u001C���ͼ�M�^���\u001F��</v�5l\t[�WBA(\u0014:�\u0011L!\u0012��)XN(\t�P\u0011*��P\u0013j\u0005#�\b;\u0005'�\u0012^\u0015JBCh\u0014jpQH���V!��\u0011:��\u0010\t1\u0005O���^�9\f\n��3�Aa�p�0(L\u001E^\u0011�0y8�\u0019\u0014&\u000Fg\u000E�����aP�<�%@a�p�\r\n���lP�<�'@a�(Q7(L\u001E%�\u0006�ɣL����Q��Aa�(��0y�\u001CΡpy� ;\u0014.�\nB���Q�P���Q�P���Q\u0001�P�<*��P�<*N�P�<*N�P�<j\u000E�P�<��Z(\\\u001Euz-\u0014.�\u000B@\u000E�ˣ�ápy�i0(\\\u001Eq�e]���.������9�뎦�O��V\u000Ec��k��S�����\u0005؍\u0006�\nendstream\r\n13 0 obj\n<< \n/Type /FontDescriptor\n/Ascent 905\n/CapHeight 0\n/Descent 212\n/Flags 32\n/ItalicAngle 0\n/StemV 0\n/FontBBox [-665 -325 2000 1006]\n/FontName /ArialMT\n>>\nendobj\n14 0 obj\n<< \n/Type /Font\n/Subtype /TrueType\n/FirstChar 32\n/LastChar 106\n/Encoding 15 0 R\n/Widths 16 0 R\n/ToUnicode 17 0 R\n/FontDescriptor 13 0 R\n/BaseFont /ArialMT\n>>\nendobj\n15 0 obj\n<< \n/Type /Encoding\n/BaseEncoding /WinAnsiEncoding\n/Differences [32 /quotedbl /quotesingle /parenleft /parenright /comma /hyphen /period /slash /zero /one /two /three /four /five /six /seven /eight /nine /colon /at /A /B /C /D /E /F /G /H /I /J /K /L /M /N /O /P /R /S /T /V /a /b /c /d /e /f /g /h /i /j /l /m /n /o /p /q /r /s /t /u /v /x /z /asciitilde /degree /aacute /acircumflex /atilde /ccedilla /eacute /ecircumflex /iacute /oacute /otilde /udieresis]\n>>\nendobj\n16 0 obj\n[355 191 333 333 278 333 278 278 556 556 556 556 556 556 556 556 556 556 278 1015 667 667 722 722 667 611 778 722 278 500 667 556 833 722 778 667 722 667 611 667 556 556 500 556 556 278 556 556 222 222 222 833 556 556 556 556 333 500 278 556 500 500 500 584 400 556 556 556 500 556 556 278 556 556 556]\nendobj\n17 0 obj\n<< \n/Filter /FlateDecode\n/Length 670\n>>\nstream\nx�U��n�P\u0014F�9O�a:\n�/N$�$� �^Ԩ\u000F`��\b�\u0018�A޾^�O#u��\u00056{�����ק�xmV�?�s�\\��x�������4��r��Ͷ\u0019����~���ܴ����z-��i<7��nu�k���:�77��_�����2\u001F����\u000F�\\?�].ʩL�fݬ��f(��o��{w*���c-���9\u000F����e���z�ov�oVe\u001A��vՆn:���m�q�X����\u00126��BK�\u0012�\nw\u0004#��=�\t��\u0013�\u0010\n\u0003!\t�P\b-�U\u0018\tw��\u001AlM�'�+l\b\u001D�S��\u001E\b\u0007\u0005#�^�\t\u0003aP\bB!\u0014�$��Q�m\r\n����Aa�0�5(L\u001E�\u0011�0y8�\u001B\u0014&\u000Fgt����nP�<��\r\n��3�Aa�pF7(L\u001E��\u0006�����P�<��\u000E�����P�<��\u000E���\u000F\u0004(L\u001EΫt(\\\u001EΫt(\\\u001EΫt(\\\u001EΫt(\\\u001E��C��\b�w(\\\u001E��\u000E��#Xߡpy\u0004�:\u0014.��ԡpyd}(\u0014.��\u000F�����P�<\u0012S����_�py$�\u000E��#1\r(B\u001E�i@\u0011�HL\u0003��GB\u0018P�<\u0012\"�\u0010\u0006\u0014!��0�\by�\u0010\u0006\u0014!���\u0003��G��\u0001Eȣe��\"�Ѳ~@\u0011�hY?�\by��\u001FP�<ں���X@�FWQ�<�Y�κڂ���6\bz����픖�?�$���ڰ�[��E���u�[��h�6���S�y/��8����}kg@�\u0017�+���\u0006\u001B�i.��{[�l��\u0006GJ�����:h���׵r�\u001B�\"��\u001B�1�t$p\r\u001E%���%�\u000BV.�s�:�괨>ͧ�\u0011\u0004\u0010�9=�̟p�+��J�H1���o�]�k��6��\u0017\u0006�\u0006\"(�\u001B���\u0016���٪0a)��\u001DMo�ڭ��X@NW�#\u0015��}c�Si\u0003bC��\u001E�՛�j\u001A}������<�8\u0015�i�ͥ��(Ko\u00038#\u001CV\u000F��h58.�$q������\u0016�\u000EgL�����\\u�\u0003:\u0002\u00141���u������;��ҤW��дgr:��4iƗ�I\r�ڭ�$���+�a{\u0001\u0004N��rH=\u0000��\u0018�0ijXa���s�\u000F��\u0011�`B\u0007ݟc]\u000E��[.�!I���\b\u0001�&�x:��s�\u0013����f�4ٜ\ \u0014QE#t���4�(��(��(��(��(��(��(��(��(��JZJ)h��(��))h��(����(��(��(��(��(��(��(��(��(��(��(��(��(��(��(��(��))i)h��(��(��(��(��(��(��(��(��(��(��(��(��(��(��(��(����(��(��(��(��))h��(���Z(��(��(��(��(��(��(��(��(��))h��(��(��(��(��(��(��(��(���Z(��(���SS�\u001A}\u0014RQKE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE%\u0014QE-\u0014��QE\u0014��QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014�QE\u0014QKE\u0014QE\u0014QE\u0014QE\u0014RR�E\u0014QE%-\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE%-%\u0014�QE\u0014QE\u0014QE\u0014�QE\u0014�QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014RQKE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QIE-\u0014QE\u0014�Ҙ�xԔQIE-\u0014QE\u0014QE\u0014�R�E\u0014QE\u0014QE\u0014QE\u0014QE%-\u0014�QE\u0014R�E\u0014QE\u0014QE%\u0014�QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014��QE\u0014RQE\u0014QKE\u0014QE\u0014QE\u0014QE\u0014RQKE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QIKIE\u0014�QE\u0014��RR�E\u0014RQKE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014RR�E\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014RR�E\u0014QE\u0014RR�E\u0014R\u001E��\u0004\u001A}\u0014RQE-\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE%\u0014QKE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014RR�E\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QIE\u0014R�E\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014RR�E\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\ \ \u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE\u0014QE��\nendstream\n\nendobj 51 0 obj\n[52 0 R]\nendobj\n52 0 obj\n<< \n/Rect [286 65 360 57]\n/Subtype /Link\n/Type /Annot\n/Border [0 0 0]\n/A << \n/S /URI\n/type /Action\n/URI (mailto:a@m.com)\n>>\n>>\nendobj\n1 0 obj\n<< \n/Type /Pages\n/Kids [43 0 R 48 0 R]\n/Count 2\n>>\nendobj\n53 0 obj\n<< \n/Type /Catalog\n/Pages 1 0 R\n>>\nendobj\n54 0 obj\n<< \n/Producer (Image Recognition Integrated Systems, Autoformat5,0,0,235)\n/Creator (I.R.I.S.)\n/CreationDate (D:20130415145343-02'00')\n>>\nendobj\nxref\n0 55\n0000000000 65535 f \n0000489463 00000 n \n0000000009 00000 n \n0000000125 00000 n \n0000000307 00000 n \n0000000490 00000 n \n0000000674 00000 n \n0000000787 00000 n \n0000001212 00000 n \n0000001389 00000 n \n0000001570 00000 n \n0000001880 00000 n \n0000002086 00000 n \n0000002658 00000 n \n0000002826 00000 n \n0000003000 00000 n \n0000003479 00000 n \n0000003798 00000 n \n0000004536 00000 n \n0000004724 00000 n \n0000004914 00000 n \n0000005153 00000 n \n0000005287 00000 n \n0000005749 00000 n \n0000005923 00000 n \n0000006103 00000 n \n0000006197 00000 n \n0000006223 00000 n \n0000006508 00000 n \n0000006681 00000 n \n0000006859 00000 n \n0000007162 00000 n \n0000007368 00000 n \n0000007940 00000 n \n0000008119 00000 n \n0000008304 00000 n \n0000008410 00000 n \n0000008452 00000 n \n0000008765 00000 n \n0000008942 00000 n \n0000009122 00000 n \n0000009225 00000 n \n0000009255 00000 n \n0000009546 00000 n \n0000009741 00000 n \n0000011327 00000 n \n0000238075 00000 n \n0000238100 00000 n \n0000238254 00000 n \n0000238449 00000 n \n0000240319 00000 n \n0000489285 00000 n \n0000489310 00000 n \n0000489529 00000 n \n0000489580 00000 n \ntrailer\n<< \n/Root 53 0 R\n/Size 55\n/Info 54 0 R\n>>\nstartxref\n489733\n%%EOF","doc_id":"6828373853754fcc8020cfcf0a24eef2","favorite_count":0,"filename":"Gravação de Imagens.pdf","host":"http://xxxx.com","label":["SGEDWEB"],"last_modified":"2023-08-23T16:29:43.893Z","mimetype":"application/pdf","role":["Rguest"],"segment":"20230823132430","site":"http://xxx.com","title":"Gravação de Imagens.pdf","title_en":"Gravação de Imagens.pdf","url":"http://xxx.com/file_download.php?file_id=63&type=bug","virtual_host":[]}

2023-08-23 13:31:21,464 [eshttp] DEBUG <<< {"took":761,"errors":false,"items":[{"index":{"_index":"fess.20230810","_id":"f2db38e8ea9775b4162b35512fae5059ca88fca5cb1294a707b2507f95d69f558dca5027372cbcb6901a99c2406b745e37c8cb84f23633e784ed3cc3b5b9093a","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":61935,"_primary_term":18,"status":201}},{"index":{"_index":"fess.20230810","_id":"216496b77d74ee334e3f1c651257234275d281daf503b7c33ecd3aa281bfa9d1647710def39270fbce6e3e5de71a863d9a4054e88f40b0d424110aa10ea89d14","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":62308,"_primary_term":18,"status":201}},{"index":{"_index":"fess.20230810","_id":"0d25ea891d2e7c52718dde74ba2e7674aab9decd6f10e8a101ea6c0cb46cb2afe3713e3a694c2632d56aadf2ad5fc1f1f212e47988451470a63e896225241e2b","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":61936,"_primary_term":18,"status":201}}]}
2023-08-23 13:31:21,465 [eshttp] DEBUG Response in Memory

shinsuke · August 24, 2023, 4:56am

The proper log message is “use {} from {}, not {}” or “use a default extractor as {} by {}”.
The placeholders are the extractor name and mime type.

shinsuke · August 24, 2023, 12:30pm

I added default_mimetype as a parameter.
You can set default_mimetype=application/pdf to Parameter.

rafael · August 24, 2023, 1:47pm

Couldnt find that log message. I set default_mimetype, but didnt worked also. Its happening with all files, xls,doc,eml… Thank you, we will keep as we did before, with apache/php we can export properly the files to a folder and then index with fess.
Thank you.