Last modified to fess-urls.log

English:Fess
1

I would like to have the last-modified time from our pages in the fess-urls.log-file. Is that possible?

If our web pages has: <meta name="last-modified" content="2026-09-04T13:42:17+02:00">

I want to put that field as last_modified in fess. Im thinking
field.xpath.last_modified=//META\[@name='last-modified'\]/@content

But is it possible to get that date in the fess-urls.log file?

The reason is I would like to create a sitemap.xml out of fess-urls.log and then I would like to have the page’s real <lastmod> , not when fess crawled it as
<url>
<loc>https://www.oru.se/sida-b/</loc>
<lastmod>2026-09-02T09:15:32+02:00</lastmod>
</url>**
**

How do i include lastmod in fess-urls.log?

2

fess-urls.log is a crawl statistics log — each line only holds the URL, when the crawl of that URL started, and per-step timings — so there is no way to add last_modified to it.

Your field.xpath idea is the right one. Put this in the Configuration Parameters of the web crawl config:

field.xpath.last_modified=//META[@name='last-modified']/@content
field.config.last_modified=overwrite

The second line is only needed if your server also sends an HTTP Last-Modified header; without it the field keeps both values. Already-indexed pages pick up the meta value when they are re-crawled, and with “Check Last Modified” enabled unchanged pages are skipped, so run one full crawl after the change.

Then build the sitemap from the index instead of the log: on Fess 15.8, set api.search.scroll=true in fess_config.properties and call GET /api/v2/documents/all. It streams url and last_modified for every indexed document as NDJSON.

3

Thanks, that is a better way to go!

We are right now on 15.6.6 so we are using v1-api. Im trying to just get the results with url and last_modified and tried:

https://<Server Name>/api/v1/documents/all?q=*.*&fields=url,last_modified&num=100

But I’m getting all the data in the result. Is it possible to just get urls and their last_modified?

4

Update! We are now on V2 so now I’m having the same question but for:
https://<Server Name>/api/v2/documents/all?q=*.*&fields=url,last_modified&num=100

Another question. How do I query to get all the content of a url in the answer. I mean all the text on each page?

5

We fixed an issue where /api/v2/documents/all could stream only the first document instead of the full result set. Could you try again on Fess 15.8 and see if that resolves it?

6

For this guery i get 0 results: https://<Server Name>/api/v2/documents/all

For this guery i get all the results:https://<Server Name>/api/v2/documents/all?q=*.*

If I have this confiquration now:

query.additional.response.fields=Occupation,PageType,Criteria,Organisation
query.additional.api.response.fields=Occupation,PageType,Criteria,Organisation
query.additional.scroll.response.fields=

Do I have to put the meta in this to?query.additional.scroll.response.fields=Occupation,PageType,Criteria,Organisation

7

Two separate things here:

  1. q is required for /api/v2/documents/all — without it you’ll get 0 results. Use q=*:* to match everything, as shown in the scroll search docs.
  2. Yes, add your custom fields to query.additional.scroll.response.fields as well: query.additional.scroll.response.fields=Occupation,PageType,Criteria,Organisation. This setting controls which fields are actually fetched from the index for scroll requests — it’s independent from query.additional.response.fields / query.additional.api.response.fields, which only control what a normal search response returns.
8

Ok, thanks!

What about this. Im trying to just get the results with only url and last_modified and tried:

https://<Server Name>/api/v2/documents/all?q=*.*&fields=url,last_modified

But I’m getting all the data in the result. Is it possible to just get urls and their last_modified?

9

fields=url,last_modified isn’t a real parameter on this endpoint, so it’s silently ignored — that’s why you’re still getting everything back.

The only per-request fields.* mechanism this API supports is fields.<name>=value (dot-separated), and that’s a search filter (matches documents where that field equals the given value), not a field-selection/projection option — same as the fields.label=public example in the docs.

There’s currently no way to select which fields come back per request. The response always contains whatever query.additional.scroll.response.fields / query.additional.api.response.fields allow on the server side. If you only want url and last_modified in the output, you’d need to filter the NDJSON stream client-side.