Robots.txt handling does not read the allow rules by default and doesn't follow standards

ossis · October 11, 2021, 8:30am

For some reason, the Fess does not read the allow rules from the robots.txt by default:

codelibs/fess-crawler/blob/master/fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/HcHttpClient.java#L224

    
      
          protected Map<String, AuthSchemeProvider> authSchemeProviderMap;
          
          
protected int connectionCheckInterval = 5; // sec
          
          
protected long idleConnectionTimeout = 60 * 1000L; // 1min
          
          
protected Pattern redirectHttpStatusPattern = Pattern.compile("[3][0-9][0-9]");
          
          
protected boolean useRobotsTxtDisallows = true;
          
          
protected boolean useRobotsTxtAllows = false;
          
          
protected CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
          
          
protected AuthCache authCache = new BasicAuthCache();
          
          
protected HttpRoutePlanner routePlanner;
          
          
protected boolean redirectsEnabled = false;
          
          
protected Lookup<CookieSpecProvider> cookieSpecRegistry;

github.com

codelibs/fess-crawler/blob/master/fess-crawler/src/main/java/org/codelibs/fess/crawler/client/http/HcHttpClient.java#L570-L578

    
      
              if (useRobotsTxtAllows) {
                  for (String urlPattern : directive.getAllows()) {
                      if (StringUtil.isNotBlank(urlPattern)) {
                          urlPattern = convertRobotsTxtPathPattern(urlPattern);
                          crawlerContext.getUrlFilter().addInclude(hostUrl + urlPattern);
                      }
                  }
              }
          }

Also in the robots.txt hangling, the longest rule should match. With Fess this is not the case:

public void test_match_google_case1() {
	urlFilter.addInclude("http://example.com/p.*");
	urlFilter.addExclude("http://example.com/.*");

	final String sessionId = "id1";
	urlFilter.init(sessionId);

	assertTrue(urlFilter.match("http://example.com/page"));
}

With kind regards,
Ossi

shinsuke · October 11, 2021, 10:11pm

I’ll change it to true.