API Call to inject list of URL's to be Crawled?

English:Fess
1

(from github.com/Outstep)
Hello,

I am not sure if this is the right place to ask this, but after a lot of searching, and reviews, I have come back around and now believe that FESS is about the best possible solution that I could ask for to go with our project.

To that end, we are setting up the basic FESS with Elasticsearch cluster which is so far proceeding well.

One question that I have is within your API documentation:

https://fess.codelibs.org/12.1/api/index.html

For our project, we have a need to be able to send FESS, via it API, lists of URL’s to start crawl upon receipt and keep adding any additional URL’s to a queue to be crawled as soon as possible so that they can be indexed and later queried by the users.

I did not see any type of REST API call that would allow us to send URL’s to be FESS crawler.

Could you please tell me where to locate this information and how to use it so that I can test it out on our development system?

Most Sincerely and have a great day,

2

(from github.com/marevol)
For that use case, I think that CsvListDataStore is a better solution.
To generate sample data, an example script is csvlistdatastore.sh.
So, steps you need to do is:

  1. Create CsvListDataStore configuration
  2. Create CSV file which contains urls
  3. Put CSV file
  4. Start Crawler via Admin API
3

(from github.com/Outstep)
Hello,

Thanks for replying to my question.

I will have to look into how this might be done since actually another program will be sending over the URL data it collects each time which could be 1 URL or up to about 30 URL’s.

I was hoping to find just a nice and simple API call that I could either send one at a time, or a block of URL’s at a time since this will all be a completely automated process.

Both programs are controlled by us and will probably be running in Docker containers and only expose needed ports.

Thanks again and have a great day,