Unified · Split
| @@ -34,7 +34,7 @@ Finally, point your browser at [localhost:9292](http://localhost:9292) (the firs · 33 unmodified lines | |||
| 34 | 34 | ||
| 35 | The `thor docs:download` command is used to download/update individual documentations (e.g. `thor docs:download html css`), or all at the same time (using the `--all` option). You can see the list of available documentations by running `thor docs:list`. | 35 | The `thor docs:download` command is used to download/update individual documentations (e.g. `thor docs:download html css`), or all at the same time (using the `--all` option). You can see the list of available documentations by running `thor docs:list`. |
| 36 | 36 | ||
| 37 | **Note:** there is currently no update mechanism other than using git to update the code and `thor docs:download` to download the latest version of the docs. To stay informed about new releases, be sure to subscribe to the [newsletter](http://eepurl.com/HnLUz). | 37 | **Note:** there is currently no update mechanism other than `git pull origin master` to update the code and `thor docs:download` to download the latest version of the docs. To stay informed about new releases, be sure to subscribe to the [newsletter](http://eepurl.com/HnLUz). |
| 38 | 38 | ||
| 39 | ## Vision | 39 | ## Vision |
| 40 | 40 | ||
| @@ -74,11 +74,11 @@ The scraper is responsible for generating the documentation and index files (met · 33 unmodified lines | |||
| 74 | There are currently two kinds of scrapers: `UrlScraper` which downloads files via HTTP and `FileScraper` which reads them from the local filesystem. They both make copies of HTML documents, recursively following links that match a given set of rules and applying all sorts of modifications along the way, in addition to building an index of the files and their metadata. Documents are parsed using [Nokogiri](http://nokogiri.org). | 74 | There are currently two kinds of scrapers: `UrlScraper` which downloads files via HTTP and `FileScraper` which reads them from the local filesystem. They both make copies of HTML documents, recursively following links that match a given set of rules and applying all sorts of modifications along the way, in addition to building an index of the files and their metadata. Documents are parsed using [Nokogiri](http://nokogiri.org). |
| 75 | 75 | ||
| 76 | Modifications made to each document include: | 76 | Modifications made to each document include: |
| 77 | * removing stuff such as the document structure (`<html>`, `<head>`, etc.), comments, empty nodes, etc. | 77 | * removing content such as the document structure (`<html>`, `<head>`, etc.), comments, empty nodes, etc. |
| 78 | * fixing links (e.g. to remove duplicates) | 78 | * fixing links (e.g. to remove duplicates) |
| 79 | * replacing all external (not scraped) URLs with their fully qualified counterpart | 79 | * replacing all external (not scraped) URLs with their fully qualified counterpart |
| 80 | * replacing all internal (scraped) URLs with their unqualified and relative counterpart | 80 | * replacing all internal (scraped) URLs with their unqualified and relative counterpart |
| 81 | * adding stuff, such as a title and link to the original document | 81 | * adding content, such as a title and link to the original document |
| 82 | 82 | ||
| 83 | These modifications are applied through a set of filters using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. Each scraper includes filters specific to its documentation, one of which is tasked with figuring out the pages' metadata. | 83 | These modifications are applied through a set of filters using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. Each scraper includes filters specific to its documentation, one of which is tasked with figuring out the pages' metadata. |
| 84 | 84 | ||