Imposible to crawling repositories files when using metadata ?

(from github.com/budiangga95)
Great work and special thanks with your fess search engine @marevol !!!
I have problem about the fess search performance to crawling a website using metadata scheme like Dspace or Eprints (Institutional Repository).

In this case, the pdf file extension and the bibliography descriptions cannot to crawling using FESS 12.1.

For example in here :

can you help me crawling this sites and get the result page with pdf document?

thanks…

(from github.com/marevol)
Try to add client.robotsTxtEnabled=false to Config Parameters.