FileSearcher Theme - Deep Folder Settings

English:Fess
1

Hello,

I am using Fess 15.9.5 with the File Search theme and have indexed a Windows SMB file server.

The folder browser on the start page works well in general, but I have a problem with large folder structures.
When a folder contains many subfolders or files, Fess displays the message that the folder is too large and that not all subfolders may be displayed.

As a result, folders further down ( not deeper ) in the hierarchy can disappear from the folder navigation.

Is there a configuration option in the File Search theme to limit the displayed folder hierarchy to a specific depth?

For example, I would like to display only three levels:

  • Server
  • Main Folder
    • 1 Subfolder
      1.1 Subfolder
      1.2 Subfolder
    • 2 Subfolder
    • 3 Subfolder
    • 4 Subfolder
    • 5 Subfolder

The Problem with many Files or Subfolders was, that the lower ones, not the deeper ones will dissapper.
It looks like this :

  • Server
  • Main Folder
    • 1 Subfolder
      1.1 Subfolder
      2.1 Subfolder
      3.1 Subfolder
      1.2 Subfolder
      2.2 Subfolder
    • 2 Subfolder

( But 3, 4, 5 are lost because of the Count )

But I would not need to browse deeper subfolders in the navigation tree.

My goal is to keep all top-level folders visible while limiting the amount of folder data that the File Search theme loads and displays.

Is there a setting for maximum folder depth, maximum subfolders, or a similar option in the File Search theme?

As an Addition, i want this just for the main Fess Search Page. If i am in a started the Search ( there the folder tree ist on the left top of the screen ) i dont want a Folder Deep Limiter.

Maybe it is a feature Request, because i think it is a likely common Problem in larger File Servers.

Thank you!

2

Thanks for the report. A depth limit wouldn’t help here. The folder tree is built from a list of document URLs that Fess caps at query.facet.fields.size.max (default 1000). In a folder with more than 1000 files, the alphabetically last sub-folders are cut off, which is why 3, 4 and 5 are missing.

The File Search theme always asked for exactly 1000, so raising that setting had no effect. This is fixed in File Search 15.9.7.

After updating the theme, set query.facet.fields.size.max in fess_config.properties to a value larger than the number of files in your biggest folder (for example 10000) and restart Fess. Larger values make each folder expansion return a larger response.

3

Hey shinsuke,

and thanks for your Reply.

But what if i haf more than mabe 100.000 files in a main folder.

That would make the “Tree Build” much slower ?

But i dont get the idea of showing the count of files in the “Main Search Window”. This is to specify the folder at the beginning, where to look for my files. There is no need of showning the counted files in this directory and i think there is no need to go very deep to specify the search.

Later, after the search, it is really nice to see how many files in which directory are found.

4

Thanks for the feedback, and sorry my first answer missed the point.

The start page tree no longer counts files. It now looks up each folder’s subfolders one by one, so all of them are listed however many files the folder holds. The tree next to the search results is unchanged and still shows counts.

This is released as File Search 15.9.8. Please update the theme and try it. If you still see the “some subfolders may not be listed” note, let us know.

5

That worked perfectly. Thanks !

If i have some other maybe “Feature Requests” is it possible to contact you ?

6

Glad it works. Yes, please post any feature requests or problems in this forum. We read it and will take a look.

7

Feature Request: Crawler Progress Indicator Based on File Count

Description

It would be very useful to have a progress indicator showing the current progress of a crawler job as a percentage in the Fess administration interface.

One possible approach would be to introduce an optional preliminary crawler that runs before the Default Crawler.

Proposed implementation:

Preliminary counting crawler

An optional crawler job that runs before the Default Crawler.

It scans all configured crawl targets and counts the total number of files to be processed.

It does not download, extract, or index any file contents.

The total file count is stored for the subsequent crawl job.

Progress tracking in the Default Crawler

A counter tracks how many files have been processed during the crawl.

The current count is compared against the total file count obtained by the preliminary crawler.

The progress percentage is calculated as:

Progress (%) = Processed Files / Total Files × 100

Progress display in the administration interface

Display the current progress as a percentage and preferably as a progress bar.

Also show the number of processed files and the estimated total number of files.

The progress should update continuously while the Default Crawler is running.

Configuration

The preliminary counting crawler should be optional and configurable. Administrators should be able to enable or disable it depending on their environment and crawl requirements.

Additional consideration

Since the number of files can change between the preliminary scan and the actual crawl, the progress percentage would be an estimate rather than an absolute guarantee. Nevertheless, it would provide administrators with a much better overview of long-running crawl jobs.

8

Feature Request: Report of Files Skipped During Crawling

Description

Fess should provide an overview of files that were discovered during crawling but could not be indexed.

Currently, files exceeding the configured defaultMaxLength limit may be skipped without providing administrators with an easily accessible overview in the administration interface.

Proposed implementation:

  • Maintain a separate list or index of files that could not be indexed.
  • Record the file path/URL, file name, file size, MIME type, timestamp and reason for skipping.
  • Include reasons such as:
    • File exceeds the configured maximum content length.
    • Password-protected or encrypted document.
    • Unsupported file format.
    • Extraction or processing error.
  • Provide a dedicated administration page to review these files.
  • Allow administrators to filter and export the list (e.g. CSV).

Additional benefit

This would make it much easier to identify missing content, adjust crawler settings and ensure that important documents are not silently excluded from the search index.

9

Feature Request: Duplicate File Detection and Visualization

Description

It would be very useful to have a built-in feature for detecting and displaying duplicate files across all configured crawl targets.

Since Fess already discovers files and collects metadata such as file names, paths, file sizes and timestamps, this information could be used as a basis for identifying potential duplicates.

The purpose of this feature is exclusively to identify, display and report duplicate files. Fess must not delete, modify, move or otherwise alter any files.

Proposed implementation:

  1. Duplicate detection

    • Identify files with identical content using file hashes (e.g. SHA-256).
    • Optionally identify potential duplicates based on matching file names and file sizes.
    • Include files from all configured crawl targets.

  2. Administration interface

    • Provide a dedicated page listing detected duplicates.
    • Group identical files together.
    • Display file name, full path, file size and last modified date.
    • Show how many copies of each file exist.
    • Allow administrators to open or navigate to the corresponding file locations.

  3. Filtering and reporting

    • Filter duplicates by file type, size, directory or crawl target.
    • Export duplicate reports as CSV or Excel.
    • Display the total number of duplicate groups and affected files.

  4. Storage overview

    • Calculate the total amount of potentially redundant storage.
    • Display the potential storage savings without performing any deletion.

Important: Read-only functionality

This feature is intended exclusively for analysis and visualization. Fess must never automatically or manually delete, modify, move, rename or otherwise manipulate the original files through this functionality.

Any file management must remain entirely outside Fess and under the control of the file server administrator.

10

Feature Request: Configurable Root Folder for Directory Navigation

Description

It would be very useful to have the option to define a logical starting point (root folder) for the directory tree displayed in the Fess search interface.

In many environments, the actual document repository is located several directory levels below the configured SMB share or network root.

For example, a typical directory structure might look like this:

\\server\share\
    └── Main Repository\
        └── Documents\
            ├── Finance
            ├── Administration
            ├── Education
            ├── Construction
            └── Infrastructure

The first directory levels are merely technical or organizational containers. They do not contain relevant documents themselves and only serve as starting points for the actual document repository.

Current behavior

The directory navigation displays these container directories as additional levels in the folder tree.

This creates unnecessary navigation steps and takes up valuable space in the sidebar. Users have to expand multiple directory levels before reaching the actual main folders.

Proposed functionality

Allow administrators to define a configurable logical root folder for the directory tree.

For example:

  • The actual SMB share remains unchanged.
  • The administrator selects the directory level at which the actual document repository begins.
  • All directory levels above this point are hidden in the user-facing folder navigation.
  • The first relevant directories are displayed directly as top-level folders.

Additional requirements

  • The feature should support different root folder definitions for different crawl configurations or SMB shares.
  • The underlying file paths and indexing structure must remain unchanged.
  • Search results should continue to reference the original file paths.
  • The setting should only affect the visual folder navigation, not crawling, indexing, permissions or access restrictions.
  • Ideally, administrators should be able to configure this directly in the crawl configuration or directory navigation settings.

Expected result

Instead of displaying technical container directories at the top of the folder tree, users immediately see the relevant main folders of the document repository.

This would provide a cleaner, flatter and more intuitive folder navigation, especially in environments with deeply nested SMB directory structures.

11

Thank you for the suggestions. They are now merged for Fess 15.10 (not backported to 15.9):

  1. Crawl progress: Admin > Crawling Info shows a “Running crawls” block with the indexed count and a percentage, based on the number of documents of the previous completed crawl. The first crawl shows the count only.
  2. Skipped files: Admin > Failure URL now also records file size, MIME type and HTTP status where available (not for timeouts or connection errors), and has a CSV download.
  3. Similar files: GET /api/v2/documents/{docId}/similar returns documents with the same content signature (MinHash), i.e. identical or near-identical content. It is not a fuzzy similarity search.
  4. Root folder: Admin > System > General > Theme Settings. Add a line such as
    filesearch.roots=smb://server/share/Documents/
    (several prefixes can be separated by “|”). The File Search tree then starts at those folders. Needs File Search theme 15.10.6 or later.