(from github.com/defensivedepth)
I see a number of the following crawler log entries:
INFO The content length (20695459 byte) is over 10485760 byte. The url is ....
Does the crawler actually download the entire file before making this determination or is it attempting to get the length through another means, like the Content-Length header?