Hello,
I noticed an issue when using AI Search with larger documents.
When asking for information that is located towards the end of the first referenced document, the LLM is unable to answer the question, although the system successfully provided the correct document in the context.
If more than one document is included in the context, this problem also persists when accessing information on any page of these latter documents.
The reason for that seems to be how the buildContext() method in AbstractLlmClient.java creates the context for the LLM to answer the user query.
This method tries to load the document content from the beginning, and stops once the maximum context size is reached. As a result, content located later in the document is often excluded.
One way to mitigate this is to increase the value of rag.llm.ollama.answer.context.max.chars in fess_config.properties, which is set to 10 000 by default.
However, when handling documents with 50+ pages, increasing this value sufficiently to include them fully is often not feasible due to hardware and performance constraints. It may also be inefficient when only a small portion of the document is relevant.
One way to approach this problem in theory is by chunking the documents and only loading the relevant chunks into the context. The semantic search plugin already incorporates chunking logic and it might be possible to use these chunks for the context with some code modifications, but results from the keyword-based search don’t contain relevant chunks.
Given this, I have the following questions:
- Is there another way to configure FESS to handle large documents better in AI Search?
- Are there any plans to introduce more advanced context construction techniques in future versions (e.g. by using chunks or something else)?
I would appreciate any insights or guidance.
Thank you!