Clean up docs directory and fix links

Jed Fox committed 7 years ago

Commit 73eab0b1 parent 676f4f67 files · +133 −135Browse files

Showing the changes to docs/Scraper-Reference.md (1 of 7 files). Show all

Unified · Split

@@ -1,12 +1,10 @@
1		−---
2		−
3	1	Table of contents:
4	2
5		−* [Overview](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#overview)
6		−* [Configuration](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#configuration)
7		− - [Attributes](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#attributes)
8		− - [Filter stacks](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-stacks)
9		− - [Filter options](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-options)
	3	+* [Overview](#overview)
	4	+* [Configuration](#configuration)
	5	+ - [Attributes](#attributes)
	6	+ - [Filter stacks](#filter-stacks)
	7	+ - [Filter options](#filter-options)
10	8
11	9	## Overview
12	10
@@ -34,51 +32,51 @@ Each URL is requested only once (case-insensitive). · 21 unmodified lines
34	32
35	33	Configuration is done via class attributes and divided into three main categories:
36	34
37		−* [Attributes](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#attributes) — essential information such as name, version, URL, etc.
38		−* [Filter stacks](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-stacks) — the list of filters that will be applied to each page.
39		−* [Filter options](https://github.com/Thibaut/devdocs/wiki/Scraper-Reference#filter-options) — the options passed to said filters.
	35	+* [Attributes](#attributes) — essential information such as name, version, URL, etc.
	36	+* [Filter stacks](#filter-stacks) — the list of filters that will be applied to each page.
	37	+* [Filter options](#filter-options) — the options passed to said filters.
40	38
41	39	Note: scrapers are located in the [`lib/docs/scrapers`](https://github.com/Thibaut/devdocs/tree/master/lib/docs/scrapers/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
42	40
43	41	### Attributes
44	42
45		−* `name` [String]
46		− Must be unique.
	43	+* `name` [String]
	44	+ Must be unique.
47	45	Defaults to the class's name.
48	46
49		−* `slug` [String]
50		− Must be unique, lowercase, and not include dashes (underscores are ok).
	47	+* `slug` [String]
	48	+ Must be unique, lowercase, and not include dashes (underscores are ok).
51	49	Defaults to `name` lowercased.
52	50
53		−* `type` [String] (required, inherited)
54		− Defines the CSS class name (`_[type]`) and custom JavaScript class (`app.views.[Type]Page`) that will be added/loaded on each page. Documentations sharing a similar structure (e.g. generated with the same tool or originating from the same website) should use the same `type` to avoid duplicating the CSS and JS.
	51	+* `type` [String] (required, inherited)
	52	+ Defines the CSS class name (`_[type]`) and custom JavaScript class (`app.views.[Type]Page`) that will be added/loaded on each page. Documentations sharing a similar structure (e.g. generated with the same tool or originating from the same website) should use the same `type` to avoid duplicating the CSS and JS.
55	53	Must include lowercase letters only.
56	54
57		−* `release` [String] (required)
	55	+* `release` [String] (required)
58	56	The version of the software at the time the scraper was last run. This is only informational and doesn't affect the scraper's behavior.
59	57
60		−* `base_url` [String] (required in `UrlScraper`)
61		− The documents' location. Only URLs _inside_ the `base_url` will be scraped. "inside" more or less means "starting with" except that `/docs` is outside `/doc` (but `/doc/` is inside).
62		− Defaults to `localhost` in `FileScraper`. _(Note: any iframe, image, or skipped link pointing to localhost will be removed by the `CleanLocalUrls` filter; the value should be overridden if the documents are available online.)_
	58	+* `base_url` [String] (required in `UrlScraper`)
	59	+ The documents' location. Only URLs _inside_ the `base_url` will be scraped. "inside" more or less means "starting with" except that `/docs` is outside `/doc` (but `/doc/` is inside).
	60	+ Defaults to `localhost` in `FileScraper`. _(Note: any iframe, image, or skipped link pointing to localhost will be removed by the `CleanLocalUrls` filter; the value should be overridden if the documents are available online.)_
63	61	Unless `root_path` is set, the root/initial URL is equal to `base_url`.
64	62
65		−* `root_path` [String] (inherited)
	63	+* `root_path` [String] (inherited)
66	64	The path from the `base_url` of the root URL.
67	65
68		−* `initial_paths` [Array] (inherited)
69		− A list of paths (from the `base_url`) to add to the initial queue. Useful for scraping isolated documents.
	66	+* `initial_paths` [Array] (inherited)
	67	+ A list of paths (from the `base_url`) to add to the initial queue. Useful for scraping isolated documents.
70	68	Defaults to `[]`. _(Note: the `root_path` is added to the array at runtime.)_
71	69
72		−* `dir` [String] (required, `FileScraper` only)
73		− The absolute path where the files are located on the local filesystem.
	70	+* `dir` [String] (required, `FileScraper` only)
	71	+ The absolute path where the files are located on the local filesystem.
74	72	_Note: `FileScraper` works exactly like `UrlScraper` (manipulating the same kind of URLs) except that it substitutes `base_url` with `dir` in order to read files instead of making HTTP requests._
75	73
76		−* `params` [Hash] (inherited, `UrlScraper` only)
77		− Query string parameters to append to every URL. (e.g. `{ format: 'raw' }` → `?format=raw`)
	74	+* `params` [Hash] (inherited, `UrlScraper` only)
	75	+ Query string parameters to append to every URL. (e.g. `{ format: 'raw' }` → `?format=raw`)
78	76	Defaults to `{}`.
79	77
80		−* `abstract` [Boolean]
81		− Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/scrapers/mdn/mdn.rb) class).
	78	+* `abstract` [Boolean]
	79	+ Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/scrapers/mdn/mdn.rb) class).
82	80	Defaults to `false`.
83	81
84	82	### Filter stacks
@@ -122,67 +120,67 @@ Additionally: · 37 unmodified lines
122	120
123	121	The filter options are stored in the `options` Hash. The Hash is inheritable (a recursive copy) and empty by default.
124	122
125		−More information about how filters work is available on the [Filter Reference](https://github.com/Thibaut/devdocs/wiki/Filter-Reference) page.
	123	+More information about how filters work is available on the [Filter Reference](./filter-reference.md) page.
126	124
127	125	* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb)
128	126
129		− - `:container` [String or Proc]
130		− A CSS selector of the container element. Everything outside of it will be removed and become unavailable to the other filters. If more than one element match the selector, the first one inside the DOM is used. If no elements match the selector, an error is raised.
131		− If the value is a Proc, it is called for each page with the filter instance as argument, and should return a selector or `nil`.
132		− The default container is the `<body>` element.
133		− _Note: links outside of the container element will not be followed by the scraper. To remove links that should be followed, use a [`CleanHtml`](https://github.com/Thibaut/devdocs/wiki/Filter-Reference#cleanhtmlfilter) filter later in the stack._
	127	+ - `:container` [String or Proc]
	128	+ A CSS selector of the container element. Everything outside of it will be removed and become unavailable to the other filters. If more than one element match the selector, the first one inside the DOM is used. If no elements match the selector, an error is raised.
	129	+ If the value is a Proc, it is called for each page with the filter instance as argument, and should return a selector or `nil`.
	130	+ The default container is the `<body>` element.
	131	+ _Note: links outside of the container element will not be followed by the scraper. To remove links that should be followed, use a [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter later in the stack._
134	132
135		−* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb)
	133	+* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb)
136	134	The following options are used to modify URLs in the pages. They are useful to remove duplicates (when the same page is accessible from multiple URLs) and fix websites that have a bunch of redirections in place (when URLs that should be scraped, aren't, because they are behind a redirection which is outside of the `base_url` — see the MDN scrapers for examples of this).
137		−
138		− - `:replace_urls` [Hash]
139		− Replaces all instances of a URL with another.
	135	+
	136	+ - `:replace_urls` [Hash]
	137	+ Replaces all instances of a URL with another.
140	138	Format: `{ 'original_url' => 'new_url' }`
141		− - `:replace_paths` [Hash]
142		− Replaces all instances of a sub-path (path from the `base_url`) with another.
	139	+ - `:replace_paths` [Hash]
	140	+ Replaces all instances of a sub-path (path from the `base_url`) with another.
143	141	Format: `{ 'original_path' => 'new_path' }`
144		− - `:fix_urls` [Proc]
	142	+ - `:fix_urls` [Proc]
145	143	Called with each URL. If the returned value is `nil`, the URL isn't modified. Otherwise the returned value is used as replacement.
146		−
	144	+
147	145	_Note: before these rules are applied, all URLs are converted to their fully qualified counterpart (http://...)._
148	146
149	147	* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb)
150	148
151	149	Internal URLs are the ones _inside_ the scraper's `base_url` ("inside" more or less means "starting with", except that `/docs` is outside `/doc`). They will be scraped unless excluded by one of the following rules. All internal URLs are converted to relative URLs inside the pages.
152	150
153		− - `:skip_links` [Boolean or Proc]
154		− If `false`, does not convert or follow any internal URL (creating a single-page documentation).
	151	+ - `:skip_links` [Boolean or Proc]
	152	+ If `false`, does not convert or follow any internal URL (creating a single-page documentation).
155	153	If the value is a Proc, it is called for each page with the filter instance as argument.
156		− - `:follow_links` [Proc]
	154	+ - `:follow_links` [Proc]
157	155	Called for page with the filter instance as argument. If the returned value is `false`, does not add internal URLs to the queue.
158		− - `:trailing_slash` [Boolean]
159		− If `true`, adds a trailing slash to all internal URLs. If `false`, removes it.
	156	+ - `:trailing_slash` [Boolean]
	157	+ If `true`, adds a trailing slash to all internal URLs. If `false`, removes it.
160	158	This is another option used to remove duplicate pages.
161		− - `:skip` [Array]
	159	+ - `:skip` [Array]
162	160	Ignores internal URLs whose sub-paths (path from the `base_url`) are in the Array (case-insensitive).
163		− - `:skip_patterns` [Array]
	161	+ - `:skip_patterns` [Array]
164	162	Ignores internal URLs whose sub-paths match any Regexp in the Array.
165		− - `:only` [Array]
	163	+ - `:only` [Array]
166	164	Ignores internal URLs whose sub-paths aren't in the Array (case-insensitive) and don't match any Regexp in `:only_patterns`.
167		− - `:only_patterns` [Array]
	165	+ - `:only_patterns` [Array]
168	166	Ignores internal URLs whose sub-paths don't match any Regexp in the Array and aren't in `:only`.
169	167
170		− If the scraper has a `root_path`, the empty and `/` paths are automatically skipped.
	168	+ If the scraper has a `root_path`, the empty and `/` paths are automatically skipped.
171	169	If `:only` or `:only_patterns` is set, the root path is automatically added to `:only`.
172	170
173		− _Note: pages can be excluded from the index based on their content using the [`Entries`](https://github.com/Thibaut/devdocs/wiki/Filter-Reference#entriesfilter) filter. However, their URLs will still be converted to relative in the other pages and trying to open them will return a 404 error. Although not ideal, this is often better than having to maintain a long list of `:skip` URLs._
	171	+ _Note: pages can be excluded from the index based on their content using the [`Entries`](./filter-reference.md#entriesfilter) filter. However, their URLs will still be converted to relative in the other pages and trying to open them will return a 404 error. Although not ideal, this is often better than having to maintain a long list of `:skip` URLs._
174	172
175	173	* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb)
176	174
177		− - `:attribution` [String] (required)
	175	+ - `:attribution` [String] (required)
178	176	An HTML string with the copyright and license information. See the other scrapers for examples.
179	177
180	178	* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb)
181	179
182		− - `:title` [String or Boolean or Proc]
183		− Unless the value is `false`, adds a title to every page.
184		− If the value is `nil`, the title is the name of the page as determined by the [`Entries`](https://github.com/Thibaut/devdocs/wiki/Filter-Reference#entriesfilter) filter. Otherwise the title is the String or the value returned by the Proc (called for each page, with the filter instance as argument). If the Proc returns `nil` or `false`, no title is added.
185		− - `:root_title` [String or Boolean]
	180	+ - `:title` [String or Boolean or Proc]
	181	+ Unless the value is `false`, adds a title to every page.
	182	+ If the value is `nil`, the title is the name of the page as determined by the [`Entries`](./filter-reference.md#entriesfilter) filter. Otherwise the title is the String or the value returned by the Proc (called for each page, with the filter instance as argument). If the Proc returns `nil` or `false`, no title is added.
	183	+ - `:root_title` [String or Boolean]
186	184	Overrides the `:title` option for the root page only.
187	185
188	186	_Note: this filter is disabled by default._