Unified · Split
| @@ -16,7 +16,7 @@ Want to contribute? Great. Please review the following guidelines carefully and · 15 unmodified lines | |||
| 16 | 16 | ||
| 17 | ## Reporting bugs | 17 | ## Reporting bugs |
| 18 | 18 | ||
| 19 | 1. Update to the most recent master release; the bug may already be fixed. | 19 | 1. Update to the most recent main release; the bug may already be fixed. |
| 20 | 2. Search for existing issues; it's possible someone has already encountered this bug. | 20 | 2. Search for existing issues; it's possible someone has already encountered this bug. |
| 21 | 3. Try to isolate the problem and include steps to reproduce it. | 21 | 3. Try to isolate the problem and include steps to reproduce it. |
| 22 | 4. Share as much information as possible (e.g. browser/OS environment, log output, stack trace, screenshots, etc.). | 22 | 4. Share as much information as possible (e.g. browser/OS environment, log output, stack trace, screenshots, etc.). |
| @@ -44,7 +44,7 @@ Use the [Trello board](https://trello.com/b/6BmTulfx/devdocs-documentation) wher · 21 unmodified lines | |||
| 44 | 44 | ||
| 45 | ## Contributing new documentations | 45 | ## Contributing new documentations |
| 46 | 46 | ||
| 47 | See the [`docs` folder](https://github.com/freeCodeCamp/devdocs/tree/master/docs) to learn how to add new documentations. | 47 | See the [`docs` folder](https://github.com/freeCodeCamp/devdocs/tree/main/docs) to learn how to add new documentations. |
| 48 | 48 | ||
| 49 | **Important:** the documentation's license must permit alteration, redistribution and commercial use, and the documented software must be released under an open source license. Feel free to get in touch if you're not sure if a documentation meets those requirements. | 49 | **Important:** the documentation's license must permit alteration, redistribution and commercial use, and the documented software must be released under an open source license. Feel free to get in touch if you're not sure if a documentation meets those requirements. |
| 50 | 50 | ||
| @@ -19,7 +19,7 @@ Verify this steps before suggeting a new feature: · 18 unmodified lines | |||
| 19 | https://github.com/freeCodeCamp/devdocs/labels/feature | 19 | https://github.com/freeCodeCamp/devdocs/labels/feature |
| 20 | 20 | ||
| 21 | - Make sure your feature fits DevDocs' vision | 21 | - Make sure your feature fits DevDocs' vision |
| 22 | https://github.com/freeCodeCamp/devdocs/blob/master/README.md#vision | 22 | https://github.com/freeCodeCamp/devdocs/blob/main/README.md#vision |
| 23 | --> | 23 | --> |
| 24 | 24 | ||
| 25 | ## Summary | 25 | ## Summary |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | |||
| 4 | <!-- Please create a draft PR when you haven't completed all steps yet upon creation of the PR. --> | 4 | <!-- Please create a draft PR when you haven't completed all steps yet upon creation of the PR. --> |
| 5 | 5 | ||
| 6 | <!-- SECTION A - Adding a new scraper --> | 6 | <!-- SECTION A - Adding a new scraper --> |
| 7 | <!-- See https://github.com/freeCodeCamp/devdocs/blob/master/.github/CONTRIBUTING.md#contributing-new-documentations --> | 7 | <!-- See https://github.com/freeCodeCamp/devdocs/blob/main/.github/CONTRIBUTING.md#contributing-new-documentations --> |
| 8 | 8 | ||
| 9 | If you’re adding a new scraper, please ensure that you have: | 9 | If you’re adding a new scraper, please ensure that you have: |
| 10 | 10 | ||
| @@ -17,7 +17,7 @@ If you’re adding a new scraper, please ensure that you have: · 6 unmodified lines | |||
| 17 | - [ ] `SOURCE`: A text file containing the URL to the page the image can be found on or the URL of the original image itself | 17 | - [ ] `SOURCE`: A text file containing the URL to the page the image can be found on or the URL of the original image itself |
| 18 | 18 | ||
| 19 | <!-- SECTION B - Updating an existing documentation to it's latest version --> | 19 | <!-- SECTION B - Updating an existing documentation to it's latest version --> |
| 20 | <!-- See https://github.com/freeCodeCamp/devdocs/blob/master/.github/CONTRIBUTING.md#updating-existing-documentations --> | 20 | <!-- See https://github.com/freeCodeCamp/devdocs/blob/main/.github/CONTRIBUTING.md#updating-existing-documentations --> |
| 21 | 21 | ||
| 22 | If you're updating an existing documentation to it's latest version, please ensure that you have: | 22 | If you're updating an existing documentation to it's latest version, please ensure that you have: |
| 23 | 23 | ||
| @@ -3,7 +3,7 @@ name: Deploy · 2 unmodified lines | |||
| 3 | on: | 3 | on: |
| 4 | push: | 4 | push: |
| 5 | branches: | 5 | branches: |
| 6 | - master | 6 | - main |
| 7 | 7 | ||
| 8 | jobs: | 8 | jobs: |
| 9 | test: | 9 | test: |
| @@ -1,4 +1,4 @@ | |||
| 1 | # [DevDocs](https://devdocs.io) — API Documentation Browser [](https://travis-ci.org/freeCodeCamp/devdocs) | 1 | # [DevDocs](https://devdocs.io) — API Documentation Browser |
| 2 | 2 | ||
| 3 | DevDocs combines multiple developer documentations in a clean and organized web UI with instant search, offline support, mobile version, dark theme, keyboard shortcuts, and more. | 3 | DevDocs combines multiple developer documentations in a clean and organized web UI with instant search, offline support, mobile version, dark theme, keyboard shortcuts, and more. |
| 4 | 4 | ||
| @@ -36,7 +36,7 @@ Finally, point your browser at [localhost:9292](http://localhost:9292) (the firs · 31 unmodified lines | |||
| 36 | 36 | ||
| 37 | The `thor docs:download` command is used to download pre-generated documentations from DevDocs's servers (e.g. `thor docs:download html css`). You can see the list of available documentations and versions by running `thor docs:list`. To update all downloaded documentations, run `thor docs:download --installed`. To download and install all documentation this project has available, run `thor docs:download --all`. | 37 | The `thor docs:download` command is used to download pre-generated documentations from DevDocs's servers (e.g. `thor docs:download html css`). You can see the list of available documentations and versions by running `thor docs:list`. To update all downloaded documentations, run `thor docs:download --installed`. To download and install all documentation this project has available, run `thor docs:download --all`. |
| 38 | 38 | ||
| 39 | **Note:** there is currently no update mechanism other than `git pull origin master` to update the code and `thor docs:download --installed` to download the latest version of the docs. To stay informed about new releases, be sure to [watch](https://github.com/freeCodeCamp/devdocs/subscription) this repository. | 39 | **Note:** there is currently no update mechanism other than `git pull origin main` to update the code and `thor docs:download --installed` to download the latest version of the docs. To stay informed about new releases, be sure to [watch](https://github.com/freeCodeCamp/devdocs/subscription) this repository. |
| 40 | 40 | ||
| 41 | Alternatively, DevDocs may be started as a Docker container: | 41 | Alternatively, DevDocs may be started as a Docker container: |
| 42 | 42 | ||
| @@ -25,8 +25,8 @@ app.templates.aboutPage = -> """ · 24 unmodified lines | |||
| 25 | <strong>Copyright 2013–2021 Thibaut Courouble and <a href="https://github.com/freeCodeCamp/devdocs/graphs/contributors">other contributors</a></strong><br> | 25 | <strong>Copyright 2013–2021 Thibaut Courouble and <a href="https://github.com/freeCodeCamp/devdocs/graphs/contributors">other contributors</a></strong><br> |
| 26 | This software is licensed under the terms of the Mozilla Public License v2.0.<br> | 26 | This software is licensed under the terms of the Mozilla Public License v2.0.<br> |
| 27 | You may obtain a copy of the source code at <a href="https://github.com/freeCodeCamp/devdocs">github.com/freeCodeCamp/devdocs</a>.<br> | 27 | You may obtain a copy of the source code at <a href="https://github.com/freeCodeCamp/devdocs">github.com/freeCodeCamp/devdocs</a>.<br> |
| 28 | For more information, see the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/COPYRIGHT">COPYRIGHT</a> | 28 | For more information, see the <a href="https://github.com/freeCodeCamp/devdocs/blob/main/COPYRIGHT">COPYRIGHT</a> |
| 29 | and <a href="https://github.com/freeCodeCamp/devdocs/blob/master/LICENSE">LICENSE</a> files. | 29 | and <a href="https://github.com/freeCodeCamp/devdocs/blob/main/LICENSE">LICENSE</a> files. |
| 30 | 30 | ||
| 31 | <h2 class="_block-heading" id="plugins">Plugins and Extensions</h2> | 31 | <h2 class="_block-heading" id="plugins">Plugins and Extensions</h2> |
| 32 | <ul> | 32 | <ul> |
| @@ -15,10 +15,10 @@ app.templates.intro = """ · 14 unmodified lines | |||
| 15 | <li>To be notified about new versions, don't forget to <a href="https://github.com/freeCodeCamp/devdocs/subscription">watch the repository</a> on GitHub. | 15 | <li>To be notified about new versions, don't forget to <a href="https://github.com/freeCodeCamp/devdocs/subscription">watch the repository</a> on GitHub. |
| 16 | <li>The <a href="https://github.com/freeCodeCamp/devdocs/issues">issue tracker</a> is the preferred channel for bug reports and | 16 | <li>The <a href="https://github.com/freeCodeCamp/devdocs/issues">issue tracker</a> is the preferred channel for bug reports and |
| 17 | feature requests. For everything else, use <a href="https://gitter.im/FreeCodeCamp/DevDocs">Gitter</a>. | 17 | feature requests. For everything else, use <a href="https://gitter.im/FreeCodeCamp/DevDocs">Gitter</a>. |
| 18 | <li>Contributions are welcome. See the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/CONTRIBUTING.md">guidelines</a>. | 18 | <li>Contributions are welcome. See the <a href="https://github.com/freeCodeCamp/devdocs/blob/main/CONTRIBUTING.md">guidelines</a>. |
| 19 | <li>DevDocs is licensed under the terms of the Mozilla Public License v2.0. For more information, | 19 | <li>DevDocs is licensed under the terms of the Mozilla Public License v2.0. For more information, |
| 20 | see the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/COPYRIGHT">COPYRIGHT</a> and | 20 | see the <a href="https://github.com/freeCodeCamp/devdocs/blob/main/COPYRIGHT">COPYRIGHT</a> and |
| 21 | <a href="https://github.com/freeCodeCamp/devdocs/blob/master/LICENSE">LICENSE</a> files. | 21 | <a href="https://github.com/freeCodeCamp/devdocs/blob/main/LICENSE">LICENSE</a> files. |
| 22 | </ol> | 22 | </ol> |
| 23 | <p>Happy coding! | 23 | <p>Happy coding! |
| 24 | </div></div> | 24 | </div></div> |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | |||
| 11 | 11 | ||
| 12 | Filters use the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. They take an HTML string or [Nokogiri](http://nokogiri.org/) node as input, optionally perform modifications and/or extract information from it, and then outputs the result. Together they form a pipeline where each filter hands its output to the next filter's input. Every documentation page passes through this pipeline before being copied on the local filesystem. | 12 | Filters use the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. They take an HTML string or [Nokogiri](http://nokogiri.org/) node as input, optionally perform modifications and/or extract information from it, and then outputs the result. Together they form a pipeline where each filter hands its output to the next filter's input. Every documentation page passes through this pipeline before being copied on the local filesystem. |
| 13 | 13 | ||
| 14 | Filters are subclasses of the [`Docs::Filter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/filter.rb) class and require a `call` method. A basic implementation looks like this: | 14 | Filters are subclasses of the [`Docs::Filter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/filter.rb) class and require a `call` method. A basic implementation looks like this: |
| 15 | 15 | ||
| 16 | ```ruby | 16 | ```ruby |
| 17 | module Docs | 17 | module Docs |
| @@ -46,7 +46,7 @@ The `call` method must return either `doc` or `html`, depending on the type of f · 28 unmodified lines | |||
| 46 | - `:path` — the page's normalized path | 46 | - `:path` — the page's normalized path |
| 47 | - `:store_path` — the path where the page will be stored (equal to `:path` with `.html` at the end) | 47 | - `:store_path` — the path where the page will be stored (equal to `:path` with `.html` at the end) |
| 48 | - `:internal_urls` — the list of distinct internal URLs found within the page | 48 | - `:internal_urls` — the list of distinct internal URLs found within the page |
| 49 | - `:entries` — the [`Entry`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/models/entry.rb) objects to add to the index | 49 | - `:entries` — the [`Entry`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/models/entry.rb) objects to add to the index |
| 50 | 50 | ||
| 51 | * `css`, `at_css`, `xpath`, `at_xpath` | 51 | * `css`, `at_css`, `xpath`, `at_xpath` |
| 52 | Shortcuts for `doc.css`, `doc.xpath`, etc. | 52 | Shortcuts for `doc.css`, `doc.xpath`, etc. |
| @@ -73,23 +73,23 @@ The `call` method must return either `doc` or `html`, depending on the type of f · 20 unmodified lines | |||
| 73 | 73 | ||
| 74 | ## Core filters | 74 | ## Core filters |
| 75 | 75 | ||
| 76 | * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside) | 76 | * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside) |
| 77 | * [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc. | 77 | * [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc. |
| 78 | * [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart | 78 | * [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart |
| 79 | * [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart | 79 | * [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart |
| 80 | * [`NormalizePathsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`) | 80 | * [`NormalizePathsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`) |
| 81 | * [`CleanLocalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only) | 81 | * [`CleanLocalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only) |
| 82 | * [`InnerHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string | 82 | * [`InnerHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/inner_html.rb) — converts the document to a string |
| 83 | * [`CleanTextFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes | 83 | * [`CleanTextFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_text.rb) — removes empty nodes |
| 84 | * [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document | 84 | * [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document |
| 85 | * [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/title.rb) — prepends the document with a title (disabled by default) | 85 | * [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/title.rb) — prepends the document with a title (disabled by default) |
| 86 | * [`EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/entries.rb) — abstract filter for extracting the page's metadata | 86 | * [`EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/entries.rb) — abstract filter for extracting the page's metadata |
| 87 | 87 | ||
| 88 | ## Custom filters | 88 | ## Custom filters |
| 89 | 89 | ||
| 90 | Scrapers can have any number of custom filters but require at least the two described below. | 90 | Scrapers can have any number of custom filters but require at least the two described below. |
| 91 | 91 | ||
| 92 | **Note:** filters are located in the [`lib/docs/filters`](https://github.com/freeCodeCamp/devdocs/tree/master/lib/docs/filters/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename. | 92 | **Note:** filters are located in the [`lib/docs/filters`](https://github.com/freeCodeCamp/devdocs/tree/main/lib/docs/filters/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename. |
| 93 | 93 | ||
| 94 | ### `CleanHtmlFilter` | 94 | ### `CleanHtmlFilter` |
| 95 | 95 | ||
| @@ -141,10 +141,10 @@ The `Entries` filter is responsible for extracting the page's metadata, represen · 45 unmodified lines | |||
| 141 | 141 | ||
| 142 | The following two models are used under the hood to represent the metadata: | 142 | The following two models are used under the hood to represent the metadata: |
| 143 | 143 | ||
| 144 | * [`Entry(name, type, path)`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/models/entry.rb) | 144 | * [`Entry(name, type, path)`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/models/entry.rb) |
| 145 | * [`Type(name, slug, count)`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/models/type.rb) | 145 | * [`Type(name, slug, count)`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/models/type.rb) |
| 146 | 146 | ||
| 147 | Each scraper must implement its own `EntriesFilter` by subclassing the [`Docs::EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/entries.rb) class. The base class already implements the `call` method and includes four methods which the subclasses can override: | 147 | Each scraper must implement its own `EntriesFilter` by subclassing the [`Docs::EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/entries.rb) class. The base class already implements the `call` method and includes four methods which the subclasses can override: |
| 148 | 148 | ||
| 149 | * `get_name` [String] | 149 | * `get_name` [String] |
| 150 | The name of the default entry (aka. the page's name). | 150 | The name of the default entry (aka. the page's name). |
| @@ -78,7 +78,7 @@ In addition to the [publicly-documented commands](https://github.com/freeCodeCam · 77 unmodified lines | |||
| 78 | 78 | ||
| 79 | ## Deploying DevDocs | 79 | ## Deploying DevDocs |
| 80 | 80 | ||
| 81 | Once docs have been uploaded via `thor docs:upload` (if applicable), you can push to the DevDocs master branch (or merge the PR containing the updates). If the Travis build succeeds, the Heroku application will be deployed automatically. | 81 | Once docs have been uploaded via `thor docs:upload` (if applicable), you can push to the DevDocs main branch (or merge the PR containing the updates). If the Travis build succeeds, the Heroku application will be deployed automatically. |
| 82 | 82 | ||
| 83 | - If you're deploying documentation updates, verify that the documentations work properly once the deploy is done. Keep in mind that you'll need to wait a few seconds for the service worker to finish caching the new assets. You should see a "DevDocs has been updated" notification appear when the caching is done, after which you need to refresh the page to see the changes. | 83 | - If you're deploying documentation updates, verify that the documentations work properly once the deploy is done. Keep in mind that you'll need to wait a few seconds for the service worker to finish caching the new assets. You should see a "DevDocs has been updated" notification appear when the caching is done, after which you need to refresh the page to see the changes. |
| 84 | - If you're deploying frontend changes, monitor [Sentry](https://sentry.io/devdocs/devdocs-js/) for new JS errors once the deploy is done. | 84 | - If you're deploying frontend changes, monitor [Sentry](https://sentry.io/devdocs/devdocs-js/) for new JS errors once the deploy is done. |
| @@ -16,7 +16,7 @@ Scrapers rely on the following libraries: · 15 unmodified lines | |||
| 16 | * [HTML::Pipeline](https://github.com/jch/html-pipeline) for applying filters | 16 | * [HTML::Pipeline](https://github.com/jch/html-pipeline) for applying filters |
| 17 | * [Nokogiri](http://nokogiri.org/) for parsing HTML | 17 | * [Nokogiri](http://nokogiri.org/) for parsing HTML |
| 18 | 18 | ||
| 19 | There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base URL with a local path before reading a file. `FileScraper` uses the placeholder `localhost` base URL by default and includes a filter to remove any URL pointing to it at the end. | 19 | There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base URL with a local path before reading a file. `FileScraper` uses the placeholder `localhost` base URL by default and includes a filter to remove any URL pointing to it at the end. |
| 20 | 20 | ||
| 21 | To be processed, a response must meet the following requirements: | 21 | To be processed, a response must meet the following requirements: |
| 22 | 22 | ||
| @@ -36,7 +36,7 @@ Configuration is done via class attributes and divided into three main categorie · 13 unmodified lines | |||
| 36 | * [Filter stacks](#filter-stacks) — the list of filters that will be applied to each page. | 36 | * [Filter stacks](#filter-stacks) — the list of filters that will be applied to each page. |
| 37 | * [Filter options](#filter-options) — the options passed to said filters. | 37 | * [Filter options](#filter-options) — the options passed to said filters. |
| 38 | 38 | ||
| 39 | **Note:** scrapers are located in the [`lib/docs/scrapers`](https://github.com/freeCodeCamp/devdocs/tree/master/lib/docs/scrapers/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename. | 39 | **Note:** scrapers are located in the [`lib/docs/scrapers`](https://github.com/freeCodeCamp/devdocs/tree/main/lib/docs/scrapers/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename. |
| 40 | 40 | ||
| 41 | ### Attributes | 41 | ### Attributes |
| 42 | 42 | ||
| @@ -76,12 +76,12 @@ Configuration is done via class attributes and divided into three main categorie · 33 unmodified lines | |||
| 76 | Defaults to `{}`. | 76 | Defaults to `{}`. |
| 77 | 77 | ||
| 78 | * `abstract` [Boolean] | 78 | * `abstract` [Boolean] |
| 79 | Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/scrapers/mdn/mdn.rb) class). | 79 | Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/scrapers/mdn/mdn.rb) class). |
| 80 | Defaults to `false`. | 80 | Defaults to `false`. |
| 81 | 81 | ||
| 82 | ### Filter stacks | 82 | ### Filter stacks |
| 83 | 83 | ||
| 84 | Each scraper has two [filter](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/filter.rb) [stacks](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/filter_stack.rb): `html_filters` and `text_filters`. They are combined into a pipeline (using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library) which causes each filter to hand its output to the next filter's input. | 84 | Each scraper has two [filter](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/filter.rb) [stacks](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/filter_stack.rb): `html_filters` and `text_filters`. They are combined into a pipeline (using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library) which causes each filter to hand its output to the next filter's input. |
| 85 | 85 | ||
| 86 | HTML filters are executed first and manipulate a parsed version of the document (a [Nokogiri](http://nokogiri.org/Nokogiri/XML/Node.html) node object), whereas text filters manipulate the document as a string. This separation avoids parsing the document multiple times. | 86 | HTML filters are executed first and manipulate a parsed version of the document (a [Nokogiri](http://nokogiri.org/Nokogiri/XML/Node.html) node object), whereas text filters manipulate the document as a string. This separation avoids parsing the document multiple times. |
| 87 | 87 | ||
| @@ -98,23 +98,23 @@ replace(index, name) # replace one filter with another (index can be a n · 10 unmodified lines | |||
| 98 | 98 | ||
| 99 | Default `html_filters`: | 99 | Default `html_filters`: |
| 100 | 100 | ||
| 101 | * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside) | 101 | * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside) |
| 102 | * [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc. | 102 | * [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc. |
| 103 | * [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart | 103 | * [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart |
| 104 | * [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart | 104 | * [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart |
| 105 | * [`NormalizePathsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`) | 105 | * [`NormalizePathsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`) |
| 106 | * [`CleanLocalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only) | 106 | * [`CleanLocalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only) |
| 107 | 107 | ||
| 108 | Default `text_filters`: | 108 | Default `text_filters`: |
| 109 | 109 | ||
| 110 | * [`InnerHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string | 110 | * [`InnerHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/inner_html.rb) — converts the document to a string |
| 111 | * [`CleanTextFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes | 111 | * [`CleanTextFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_text.rb) — removes empty nodes |
| 112 | * [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document | 112 | * [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document |
| 113 | 113 | ||
| 114 | Additionally: | 114 | Additionally: |
| 115 | 115 | ||
| 116 | * [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/title.rb) is a core HTML filter, disabled by default, which prepends the document with a title (`<h1>`). | 116 | * [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/title.rb) is a core HTML filter, disabled by default, which prepends the document with a title (`<h1>`). |
| 117 | * [`EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/entries.rb) is an abstract HTML filter that each scraper must implement and responsible for extracting the page's metadata. | 117 | * [`EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/entries.rb) is an abstract HTML filter that each scraper must implement and responsible for extracting the page's metadata. |
| 118 | 118 | ||
| 119 | ### Filter options | 119 | ### Filter options |
| 120 | 120 | ||
| @@ -122,7 +122,7 @@ The filter options are stored in the `options` Hash. The Hash is inheritable (a · 1 unmodified line | |||
| 122 | 122 | ||
| 123 | More information about how filters work is available on the [Filter Reference](./filter-reference.md) page. | 123 | More information about how filters work is available on the [Filter Reference](./filter-reference.md) page. |
| 124 | 124 | ||
| 125 | * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/container.rb) | 125 | * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/container.rb) |
| 126 | 126 | ||
| 127 | - `:container` [String or Proc] | 127 | - `:container` [String or Proc] |
| 128 | A CSS selector of the container element. Everything outside of it will be removed and become unavailable to the other filters. If more than one element match the selector, the first one inside the DOM is used. If no elements match the selector, an error is raised. | 128 | A CSS selector of the container element. Everything outside of it will be removed and become unavailable to the other filters. If more than one element match the selector, the first one inside the DOM is used. If no elements match the selector, an error is raised. |
| @@ -130,7 +130,7 @@ More information about how filters work is available on the [Filter Reference](. · 1 unmodified line | |||
| 130 | The default container is the `<body>` element. | 130 | The default container is the `<body>` element. |
| 131 | _Note: links outside of the container element will not be followed by the scraper. To remove links that should be followed, use a [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter later in the stack._ | 131 | _Note: links outside of the container element will not be followed by the scraper. To remove links that should be followed, use a [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter later in the stack._ |
| 132 | 132 | ||
| 133 | * [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) | 133 | * [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/normalize_urls.rb) |
| 134 | The following options are used to modify URLs in the pages. They are useful to remove duplicates (when the same page is accessible from multiple URLs) and fix websites that have a bunch of redirections in place (when URLs that should be scraped, aren't, because they are behind a redirection which is outside of the `base_url` — see the MDN scrapers for examples of this). | 134 | The following options are used to modify URLs in the pages. They are useful to remove duplicates (when the same page is accessible from multiple URLs) and fix websites that have a bunch of redirections in place (when URLs that should be scraped, aren't, because they are behind a redirection which is outside of the `base_url` — see the MDN scrapers for examples of this). |
| 135 | 135 | ||
| 136 | - `:replace_urls` [Hash] | 136 | - `:replace_urls` [Hash] |
| @@ -144,7 +144,7 @@ More information about how filters work is available on the [Filter Reference](. · 7 unmodified lines | |||
| 144 | 144 | ||
| 145 | _Note: before these rules are applied, all URLs are converted to their fully qualified counterpart (http://...)._ | 145 | _Note: before these rules are applied, all URLs are converted to their fully qualified counterpart (http://...)._ |
| 146 | 146 | ||
| 147 | * [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) | 147 | * [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/internal_urls.rb) |
| 148 | 148 | ||
| 149 | Internal URLs are the ones _inside_ the scraper's `base_url` ("inside" more or less means "starting with", except that `/docs` is outside `/doc`). They will be scraped unless excluded by one of the following rules. All internal URLs are converted to relative URLs inside the pages. | 149 | Internal URLs are the ones _inside_ the scraper's `base_url` ("inside" more or less means "starting with", except that `/docs` is outside `/doc`). They will be scraped unless excluded by one of the following rules. All internal URLs are converted to relative URLs inside the pages. |
| 150 | 150 | ||
| @@ -170,12 +170,12 @@ More information about how filters work is available on the [Filter Reference](. · 19 unmodified lines | |||
| 170 | 170 | ||
| 171 | _Note: pages can be excluded from the index based on their content using the [`Entries`](./filter-reference.md#entriesfilter) filter. However, their URLs will still be converted to relative in the other pages and trying to open them will return a 404 error. Although not ideal, this is often better than having to maintain a long list of `:skip` URLs._ | 171 | _Note: pages can be excluded from the index based on their content using the [`Entries`](./filter-reference.md#entriesfilter) filter. However, their URLs will still be converted to relative in the other pages and trying to open them will return a 404 error. Although not ideal, this is often better than having to maintain a long list of `:skip` URLs._ |
| 172 | 172 | ||
| 173 | * [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/attribution.rb) | 173 | * [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/attribution.rb) |
| 174 | 174 | ||
| 175 | - `:attribution` [String] **(required)** | 175 | - `:attribution` [String] **(required)** |
| 176 | An HTML string with the copyright and license information. See the other scrapers for examples. | 176 | An HTML string with the copyright and license information. See the other scrapers for examples. |
| 177 | 177 | ||
| 178 | * [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/title.rb) | 178 | * [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/title.rb) |
| 179 | 179 | ||
| 180 | - `:title` [String or Boolean or Proc] | 180 | - `:title` [String or Boolean or Proc] |
| 181 | Unless the value is `false`, adds a title to every page. | 181 | Unless the value is `false`, adds a title to every page. |