I am exploring Fess to see if it can work for a project that is being worked on at the moment and I really like what I have seen in Fess.
Actually, I am not completely sure if this is a Fess question or Elasticsearch question since I am researching the best approach to make it work which might require a plugin of some type to be written.
We have some JSON and/or XML URL data feeds coming from a type of metasearch engine server. Basically, that data source is sent a query and it returns a list of URL’s along with supporting fields like descriptions, date, etc…
I would like to be able to have Fess send the query to these metasearch engines, receive and aggregate the results, after which they would be stored in the Fess Elasticsearch distributed storage data servers so that on the next similar query, Fess to look to see if it already had some of the data available. If it did then it could aggregate those results with the newly returned data from the metasearch engines and return the results to the front end.
One of the clustering components that I have also been exploring for use with Fess and/or Elasticsearch is called Carrot2 which clusters results, but for now I am just trying to see how I might go about having Fess or Elasticsearch call out to the external metasearch engines and aggregate the results with what it has already, if any of this might be possible at all.
Any ideas or suggestions would be sincerely appreciated