Update URLs to github.com/freeCodeCamp/devdocs

Simon Legner committed 5 years ago

Commit 8f236e51 parent 89d53622 files · +37 −37Browse files

Unified · Split

2 files

docs/filter-reference.md +17 −17
docs/scraper-reference.md +20 −20

@@ -11,7 +11,7 @@ · 10 unmodified lines
11	11
12	12	Filters use the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. They take an HTML string or [Nokogiri](http://nokogiri.org/) node as input, optionally perform modifications and/or extract information from it, and then outputs the result. Together they form a pipeline where each filter hands its output to the next filter's input. Every documentation page passes through this pipeline before being copied on the local filesystem.
13	13
14		−Filters are subclasses of the [`Docs::Filter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter.rb) class and require a `call` method. A basic implementation looks like this:
	14	+Filters are subclasses of the [`Docs::Filter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/filter.rb) class and require a `call` method. A basic implementation looks like this:
15	15
16	16	```ruby
17	17	module Docs
@@ -46,7 +46,7 @@ The `call` method must return either `doc` or `html`, depending on the type of f · 28 unmodified lines
46	46	- `:path` — the page's normalized path
47	47	- `:store_path` — the path where the page will be stored (equal to `:path` with `.html` at the end)
48	48	- `:internal_urls` — the list of distinct internal URLs found within the page
49		− - `:entries` — the [`Entry`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/entry.rb) objects to add to the index
	49	+ - `:entries` — the [`Entry`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/models/entry.rb) objects to add to the index
50	50
51	51	* `css`, `at_css`, `xpath`, `at_xpath`
52	52	Shortcuts for `doc.css`, `doc.xpath`, etc.
@@ -73,23 +73,23 @@ The `call` method must return either `doc` or `html`, depending on the type of f · 20 unmodified lines
73	73
74	74	## Core filters
75	75
76		−* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
77		−* [`CleanHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
78		−* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
79		−* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
80		−* [`NormalizePathsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
81		−* [`CleanLocalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
82		−* [`InnerHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
83		−* [`CleanTextFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
84		−* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
85		−* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb) — prepends the document with a title (disabled by default)
86		−* [`EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) — abstract filter for extracting the page's metadata
	76	+* [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
	77	+* [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
	78	+* [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
	79	+* [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
	80	+* [`NormalizePathsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
	81	+* [`CleanLocalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
	82	+* [`InnerHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
	83	+* [`CleanTextFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
	84	+* [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
	85	+* [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/title.rb) — prepends the document with a title (disabled by default)
	86	+* [`EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/entries.rb) — abstract filter for extracting the page's metadata
87	87
88	88	## Custom filters
89	89
90	90	Scrapers can have any number of custom filters but require at least the two described below.
91	91
92		−Note: filters are located in the [`lib/docs/filters`](https://github.com/Thibaut/devdocs/tree/master/lib/docs/filters/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
	92	+Note: filters are located in the [`lib/docs/filters`](https://github.com/freeCodeCamp/devdocs/tree/master/lib/docs/filters/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
93	93
94	94	### `CleanHtmlFilter`
95	95
@@ -141,10 +141,10 @@ The `Entries` filter is responsible for extracting the page's metadata, represen · 45 unmodified lines
141	141
142	142	The following two models are used under the hood to represent the metadata:
143	143
144		−* [`Entry(name, type, path)`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/entry.rb)
145		−* [`Type(name, slug, count)`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/type.rb)
	144	+* [`Entry(name, type, path)`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/models/entry.rb)
	145	+* [`Type(name, slug, count)`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/models/type.rb)
146	146
147		−Each scraper must implement its own `EntriesFilter` by subclassing the [`Docs::EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) class. The base class already implements the `call` method and includes four methods which the subclasses can override:
	147	+Each scraper must implement its own `EntriesFilter` by subclassing the [`Docs::EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/entries.rb) class. The base class already implements the `call` method and includes four methods which the subclasses can override:
148	148
149	149	* `get_name` [String]
150	150	The name of the default entry (aka. the page's name).

docs/scraper-reference.md+20−20

@@ -16,7 +16,7 @@ Scrapers rely on the following libraries: · 15 unmodified lines
16	16	* [HTML::Pipeline](https://github.com/jch/html-pipeline) for applying filters
17	17	* [Nokogiri](http://nokogiri.org/) for parsing HTML
18	18
19		−There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base URL with a local path before reading a file. `FileScraper` uses the placeholder `localhost` base URL by default and includes a filter to remove any URL pointing to it at the end.
	19	+There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base URL with a local path before reading a file. `FileScraper` uses the placeholder `localhost` base URL by default and includes a filter to remove any URL pointing to it at the end.
20	20
21	21	To be processed, a response must meet the following requirements:
22	22
@@ -36,7 +36,7 @@ Configuration is done via class attributes and divided into three main categorie · 13 unmodified lines
36	36	* [Filter stacks](#filter-stacks) — the list of filters that will be applied to each page.
37	37	* [Filter options](#filter-options) — the options passed to said filters.
38	38
39		−Note: scrapers are located in the [`lib/docs/scrapers`](https://github.com/Thibaut/devdocs/tree/master/lib/docs/scrapers/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
	39	+Note: scrapers are located in the [`lib/docs/scrapers`](https://github.com/freeCodeCamp/devdocs/tree/master/lib/docs/scrapers/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
40	40
41	41	### Attributes
42	42
@@ -76,12 +76,12 @@ Configuration is done via class attributes and divided into three main categorie · 33 unmodified lines
76	76	Defaults to `{}`.
77	77
78	78	* `abstract` [Boolean]
79		− Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/scrapers/mdn/mdn.rb) class).
	79	+ Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/scrapers/mdn/mdn.rb) class).
80	80	Defaults to `false`.
81	81
82	82	### Filter stacks
83	83
84		−Each scraper has two [filter](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter.rb) [stacks](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter_stack.rb): `html_filters` and `text_filters`. They are combined into a pipeline (using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library) which causes each filter to hand its output to the next filter's input.
	84	+Each scraper has two [filter](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/filter.rb) [stacks](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/core/filter_stack.rb): `html_filters` and `text_filters`. They are combined into a pipeline (using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library) which causes each filter to hand its output to the next filter's input.
85	85
86	86	HTML filters are executed first and manipulate a parsed version of the document (a [Nokogiri](http://nokogiri.org/Nokogiri/XML/Node.html) node object), whereas text filters manipulate the document as a string. This separation avoids parsing the document multiple times.
87	87
@@ -98,23 +98,23 @@ replace(index, name) # replace one filter with another (index can be a n · 10 unmodified lines
98	98
99	99	Default `html_filters`:
100	100
101		−* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
102		−* [`CleanHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
103		−* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
104		−* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
105		−* [`NormalizePathsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
106		−* [`CleanLocalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
	101	+* [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
	102	+* [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
	103	+* [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
	104	+* [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
	105	+* [`NormalizePathsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
	106	+* [`CleanLocalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
107	107
108	108	Default `text_filters`:
109	109
110		−* [`InnerHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
111		−* [`CleanTextFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
112		−* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
	110	+* [`InnerHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
	111	+* [`CleanTextFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
	112	+* [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
113	113
114	114	Additionally:
115	115
116		−* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb) is a core HTML filter, disabled by default, which prepends the document with a title (`<h1>`).
117		−* [`EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) is an abstract HTML filter that each scraper must implement and responsible for extracting the page's metadata.
	116	+* [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/title.rb) is a core HTML filter, disabled by default, which prepends the document with a title (`<h1>`).
	117	+* [`EntriesFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/entries.rb) is an abstract HTML filter that each scraper must implement and responsible for extracting the page's metadata.
118	118
119	119	### Filter options
120	120
@@ -122,7 +122,7 @@ The filter options are stored in the `options` Hash. The Hash is inheritable (a · 1 unmodified line
122	122
123	123	More information about how filters work is available on the [Filter Reference](./filter-reference.md) page.
124	124
125		−* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb)
	125	+* [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/container.rb)
126	126
127	127	- `:container` [String or Proc]
128	128	A CSS selector of the container element. Everything outside of it will be removed and become unavailable to the other filters. If more than one element match the selector, the first one inside the DOM is used. If no elements match the selector, an error is raised.
@@ -130,7 +130,7 @@ More information about how filters work is available on the [Filter Reference](. · 1 unmodified line
130	130	The default container is the `<body>` element.
131	131	_Note: links outside of the container element will not be followed by the scraper. To remove links that should be followed, use a [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter later in the stack._
132	132
133		−* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb)
	133	+* [`NormalizeUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb)
134	134	The following options are used to modify URLs in the pages. They are useful to remove duplicates (when the same page is accessible from multiple URLs) and fix websites that have a bunch of redirections in place (when URLs that should be scraped, aren't, because they are behind a redirection which is outside of the `base_url` — see the MDN scrapers for examples of this).
135	135
136	136	- `:replace_urls` [Hash]
@@ -144,7 +144,7 @@ More information about how filters work is available on the [Filter Reference](. · 7 unmodified lines
144	144
145	145	_Note: before these rules are applied, all URLs are converted to their fully qualified counterpart (http://...)._
146	146
147		−* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb)
	147	+* [`InternalUrlsFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb)
148	148
149	149	Internal URLs are the ones _inside_ the scraper's `base_url` ("inside" more or less means "starting with", except that `/docs` is outside `/doc`). They will be scraped unless excluded by one of the following rules. All internal URLs are converted to relative URLs inside the pages.
150	150
@@ -170,12 +170,12 @@ More information about how filters work is available on the [Filter Reference](. · 19 unmodified lines
170	170
171	171	_Note: pages can be excluded from the index based on their content using the [`Entries`](./filter-reference.md#entriesfilter) filter. However, their URLs will still be converted to relative in the other pages and trying to open them will return a 404 error. Although not ideal, this is often better than having to maintain a long list of `:skip` URLs._
172	172
173		−* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb)
	173	+* [`AttributionFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/attribution.rb)
174	174
175	175	- `:attribution` [String] (required)
176	176	An HTML string with the copyright and license information. See the other scrapers for examples.
177	177
178		−* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb)
	178	+* [`TitleFilter`](https://github.com/freeCodeCamp/devdocs/blob/master/lib/docs/filters/core/title.rb)
179	179
180	180	- `:title` [String or Boolean or Proc]
181	181	Unless the value is `false`, adds a title to every page.