Unified · Split
| @@ -10,7 +10,7 @@ · 9 unmodified lines | |||
| 10 | 10 | ||
| 11 | ## Overview | 11 | ## Overview |
| 12 | 12 | ||
| 13 | Starting from a root url, scrapers recursively follow links that match a set of rules, passing each valid response through a chain of filters before writing the file on the local filesystem. They also create an index of the pages' metadata (determined by one filter), which is dumped into a JSON file at the end. | 13 | Starting from a root URL, scrapers recursively follow links that match a set of rules, passing each valid response through a chain of filters before writing the file on the local filesystem. They also create an index of the pages' metadata (determined by one filter), which is dumped into a JSON file at the end. |
| 14 | 14 | ||
| 15 | Scrapers rely on the following libraries: | 15 | Scrapers rely on the following libraries: |
| 16 | 16 | ||
| @@ -18,13 +18,13 @@ Scrapers rely on the following libraries: · 1 unmodified line | |||
| 18 | * [HTML::Pipeline](https://github.com/jch/html-pipeline) for applying filters | 18 | * [HTML::Pipeline](https://github.com/jch/html-pipeline) for applying filters |
| 19 | * [Nokogiri](http://nokogiri.org/) for parsing HTML | 19 | * [Nokogiri](http://nokogiri.org/) for parsing HTML |
| 20 | 20 | ||
| 21 | There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base url with a local path before reading a file. | 21 | There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base URL with a local path before reading a file. `FileScraper` uses the placeholder `localhost` base URL by default and includes a filter to remove any URL pointing to it at the end. |
| 22 | 22 | ||
| 23 | To be processed, a response must meet the following requirements: | 23 | To be processed, a response must meet the following requirements: |
| 24 | 24 | ||
| 25 | * 200 status code | 25 | * 200 status code |
| 26 | * HTML content type | 26 | * HTML content type |
| 27 | * effective URL (after redirection) contained in the base url (explained below) | 27 | * effective URL (after redirection) contained in the base URL (explained below) |
| 28 | 28 | ||
| 29 | (`FileScraper` only checks if the file exists and is not empty.) | 29 | (`FileScraper` only checks if the file exists and is not empty.) |
| 30 | 30 | ||
| @@ -34,7 +34,7 @@ Each URL is requested only once (case-insensitive). · 3 unmodified lines | |||
| 34 | 34 | ||
| 35 | Configuration is done via class attributes and divided into three main categories: | 35 | Configuration is done via class attributes and divided into three main categories: |
| 36 | 36 | ||
| 37 | * [Attributes](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#attributes) — essential information such as name, version, url, etc. | 37 | * [Attributes](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#attributes) — essential information such as name, version, URL, etc. |
| 38 | * [Filter stacks](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-stacks) — the list of filters that will be applied to each page. | 38 | * [Filter stacks](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-stacks) — the list of filters that will be applied to each page. |
| 39 | * [Filter options](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-options) — the options passed to said filters. | 39 | * [Filter options](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-options) — the options passed to said filters. |
| 40 | 40 | ||
| @@ -59,6 +59,7 @@ Configuration is done via class attributes and divided into three main categorie · 18 unmodified lines | |||
| 59 | 59 | ||
| 60 | * `base_url` [String] **(required in `UrlScraper`)** | 60 | * `base_url` [String] **(required in `UrlScraper`)** |
| 61 | The documents' location. Only URLs _inside_ the `base_url` will be scraped. "inside" more or less means "starting with" except that `/docs` is outside `/doc` (but `/doc/` is inside). | 61 | The documents' location. Only URLs _inside_ the `base_url` will be scraped. "inside" more or less means "starting with" except that `/docs` is outside `/doc` (but `/doc/` is inside). |
| 62 | `FileScraper`'s default is `localhost`. (Note: any iframe, image, or skipped link pointing to localhost will be removed by the `CleanLocalUrls` filter; the value should be overridden if the documents are available online.) | ||
| 62 | Unless `root_path` is set, the root/initial URL is equal to `base_url`. | 63 | Unless `root_path` is set, the root/initial URL is equal to `base_url`. |
| 63 | 64 | ||
| 64 | * `root_path` [String] **(inherited)** | 65 | * `root_path` [String] **(inherited)** |