# Crawling with references to other sites

**URL:** <https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616>\
**Category:** English:Fess\
**Created:** [December 12, 2016, 3:14pm UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616 "2016-12-12T15:14:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![discuss](https://discuss.codelibs.org/user_avatar/discuss.codelibs.org/discuss/32/298_2.png) [@discuss](https://discuss.codelibs.org/u/discuss)\
**Post date:** [December 12, 2016, 3:14pm UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616/1 "2016-12-12T15:14:29Z")

</div>

(from [github.com/pcolmer](http://github.com/pcolmer))  
I want to set up Fess with multiple crawlers where each crawler looks after a single web site with a label to match (thus making it easy to filter search results by web site).

However, some of our web sites link to PDFs that are hosted on other web sites. Is there a way that I can easily get those PDFs included in the search results?

Thanks.

---

<div class="post-metadata">

**Author:** ![discuss](https://discuss.codelibs.org/user_avatar/discuss.codelibs.org/discuss/32/298_2.png) [@discuss](https://discuss.codelibs.org/u/discuss)\
**Post date:** [December 12, 2016, 9:27pm UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616/2 "2016-12-12T21:27:50Z")

</div>

(from [github.com/marevol](http://github.com/marevol))  
You can use Included Paths for Label(do not specify labels in web crawling config).  
[http://fess.codelibs.org/10.3/admin/labeltype-guide.html#included-paths](http://fess.codelibs.org/10.3/admin/labeltype-guide.html#included-paths)

---

<div class="post-metadata">

**Author:** ![discuss](https://discuss.codelibs.org/user_avatar/discuss.codelibs.org/discuss/32/298_2.png) [@discuss](https://discuss.codelibs.org/u/discuss)\
**Post date:** [December 13, 2016, 10:23am UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616/3 "2016-12-13T10:23:43Z")

</div>

(from [github.com/pcolmer](http://github.com/pcolmer))  
So if I’ve got something like:

[https://www.96boards.org/.\*](https://www.96boards.org/.*)

as the primary path to be included, can I then have:

```
.*/.*\.pdf

```

to have fess include PDFs that it finds from any referrals from [www.96boards.org](http://www.96boards.org)? Or do I need to be explicit for the URL?

---

<div class="post-metadata">

**Author:** ![discuss](https://discuss.codelibs.org/user_avatar/discuss.codelibs.org/discuss/32/298_2.png) [@discuss](https://discuss.codelibs.org/u/discuss)\
**Post date:** [December 13, 2016, 10:24am UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616/4 "2016-12-13T10:24:28Z")

</div>

(from [github.com/pcolmer](http://github.com/pcolmer))  
When you say “do not specify labels in web crawling config”, is that because configuring the labels with the paths to be included is the correct way to do it? It is a little bit confusing because you _can_ specify labels in the web crawling config, so I’m trying to understand the best/correct way to do things.

---

<div class="post-metadata">

**Author:** ![discuss](https://discuss.codelibs.org/user_avatar/discuss.codelibs.org/discuss/32/298_2.png) [@discuss](https://discuss.codelibs.org/u/discuss)\
**Post date:** [December 13, 2016, 1:21pm UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616/5 "2016-12-13T13:21:26Z")

</div>

(from [github.com/marevol](http://github.com/marevol))  
I might miss something…  
If you want to get PDF in a search result, use Facet search(right side menu: FIle Type):  
[https://search.n2sm.co.jp/search/?q=test&ex\_q=filetype%3Apdf](https://search.n2sm.co.jp/search/?q=test&ex_q=filetype%3Apdf)  
This does not require label or other settings.

If you want to categorize search results, Label is useful.  
There is 2 way to set labels to indexed documents:

- Select labels in Crawling Config: [http://fess.codelibs.org/10.3/admin/webconfig-guide.html#labels](http://fess.codelibs.org/10.3/admin/webconfig-guide.html#labels)
- Specify url pattern in Label Setting: [http://fess.codelibs.org/10.3/admin/labeltype-guide.html#included-paths](http://fess.codelibs.org/10.3/admin/labeltype-guide.html#included-paths)

---

<div class="post-metadata">

**Author:** ![discuss](https://discuss.codelibs.org/user_avatar/discuss.codelibs.org/discuss/32/298_2.png) [@discuss](https://discuss.codelibs.org/u/discuss)\
**Post date:** [December 13, 2016, 1:24pm UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616/6 "2016-12-13T13:24:53Z")

</div>

(from [pcolmer (Philip Colmer) · GitHub](http://github.com/pcolmer))

> If you want to get PDF in a search result, use Facet search(right side menu: FIle Type):  
> [test-Fess](https://search.n2sm.co.jp/search/?q=test&ex_q=filetype%3Apdf)  
> This does not require label or other settings.

The challenge I’ve got is that the PDFs are not hosted on the web site that is being crawled. So I’m thinking that having:

```
https://www.96boards.org/.*
*/.*\.pdf

```

might get the crawler to retrieve and then index any PDFs - stored on any web site - that are referenced from pages on [www.96boards.org](http://www.96boards.org).

Looking at the fess log, it looks like those paths are working … except that the PDFs are not being retrieved from Elasticsearch when I try to search for them using words that I know appear in the PDF. Do I need to put something into Included URLs For Indexing as well, or should the system be indexing everything it retrieves?

---

<div class="post-metadata">

**Author:** ![discuss](https://discuss.codelibs.org/user_avatar/discuss.codelibs.org/discuss/32/298_2.png) [@discuss](https://discuss.codelibs.org/u/discuss)\
**Post date:** [December 13, 2016, 1:35pm UTC](https://discuss.codelibs.org/t/crawling-with-references-to-other-sites/616/7 "2016-12-13T13:35:28Z")

</div>

(from [github.com/pcolmer](http://github.com/pcolmer))  
I think this is working as I need it to be. Thanks for creating such a great platform!
