Fix: Fess-ds-wikipedia: wikitext markup leaks into indexed content.

[ram-fabian-heinrich]

With source=xml crawls, WikiTextParser.getPlainText() leaves
wikitext markup in the indexed content and digest fields. Section headings (== Heading ==), list markers (*, #), nested templates and infoboxes, multi-line comments and refs, and image captions all show up as raw markup in search results. It also removes every apostrophe (“Einstein’s” becomes “Einsteins”). This affects any MediaWiki XML export, including the official *-pages-articles.xml.bz2 dumps.

I’ve opened two pull requests that fix this in different ways, so you can choose whichever suits the project:

• #24 ( fix: strip section-header and list-marker wikitext from getPlainText() by ram-fabian-heinrich · Pull Request #24 · codelibs/fess-ds-wikipedia · GitHub ):
  small regex fix. It extends the existing regex chain to handle headings, list markers, comments, variants and piped links. No new dependencies.
• #25 ( feat: extract plain text with the Sweble wikitext parser by ram-fabian-heinrich · Pull Request #25 · codelibs/fess-ds-wikipedia · GitHub ):
  real wikitext parser. It replaces the regex chain with the Sweble parser (Apache-2.0), bundled into the jar the way fess-ds-git does it. On three real Wikipedia articles no markup is left, against 34–402 leftovers with the current regex. The costs are a larger jar (about 3.8 MB) and slower parsing (roughly 1.5–6×).

Motivation

We index our internal MediaWiki through its XML export, and search result snippets regularly show raw wiki syntax. Regex
fixes can’t handle nested structures such as templates inside templates, which is why I also proposed the parser-based version. I am going to test the parser version in production, and I’m happy to adjust either PR.

[shinsuke]

Thanks for the report and both pull requests!

We merged #25 (Sweble parser) and a follow-up, #26, which also includes your regex fixes from #24. #26 adds:

• Namespace names read from the dump’s , so images and categories are handled on non-English and custom-named wikis (e.g. [[ファイル:…]], [[Kategorie:…]])

• URLs are kept in the content

• Only the JAXB API is bundled, relocated (jar: 3.8 MB → 2.5 MB)

• A new text_extractor=sweble|regex parameter (regex is faster and serves as a rollback)

For Wikipedia and other Wikimedia wikis, we recommend source=cirrus (CirrusSearch dumps), since they contain text already rendered by MediaWiki.

[ram-fabian-heinrich]

Thank you for addressing this so fast! For initial testing the xml export seemed to be the easiest and least invasive. For smaller wikis an additional cirrus / OpenSearch instance also felt like too much if fess could then just handle the whole search. For the future cirrus sounds promising and I also understand that users who are interested in fess will most likely already have set up cirrus. I will take a closer look at cirrus soon