krisyotam/devdocs

.editorconfig+3−0

@@ -6,3 +6,6 @@ indent_style = space · 5 unmodified lines
6	indent_size = 2	6	indent_size = 2
7	insert_final_newline = true	7	insert_final_newline = true
8	trim_trailing_whitespace = true	8	trim_trailing_whitespace = true
		9
		10	[*.md]
		11	trim_trailing_whitespace = false

.github/CONTRIBUTING.md+7−17

 ## Requesting new features
 1. Search for similar feature requests; someone may have already requested it.
-2. Make sure your feature fits DevDocs's [vision](https://github.com/freeCodeCamp/devdocs/blob/master/README.md#vision).
+2. Make sure your feature fits DevDocs's [vision](../README.md#vision).
 3. Provide a clear and detailed explanation of the feature and why it's important to add it.
-For general feedback and ideas, please use the [mailing list](https://groups.google.com/d/forum/devdocs).
 ## Requesting new documentations
 Please don't open issues to request new documentations.
 Use the [Trello board](https://trello.com/b/6BmTulfx/devdocs-documentation) where everyone can vote.
 ## Contributing code and features
 1. Search for existing issues; someone may already be working on a similar feature.
-2. Before embarking on any significant pull request, please open an issue describing the changes you intend to make. Otherwise you risk spending a lot of time working on something I may not want to merge. This also tells other contributors that you're working on the feature.
+2. Before embarking on any significant pull request, please open an issue describing the changes you intend to make. Otherwise you risk spending a lot of time working on something we may not want to merge. This also tells other contributors that you're working on the feature.
 3. Follow the [coding conventions](#coding-conventions).
 4. If you're modifying the Ruby code, include tests and ensure they pass.
 5. Try to keep your pull request small and simple.
 ## Contributing new documentations
-See the [wiki](https://github.com/freeCodeCamp/devdocs/wiki) to learn how to add new documentations.
+See the [`docs` folder](https://github.com/freeCodeCamp/devdocs/tree/master/docs) to learn how to add new documentations.
 **Important:** the documentation's license must permit alteration, redistribution and commercial use, and the documented software must be released under an open source license. Feel free to get in touch if you're not sure if a documentation meets those requirements.
 * Remove as much content and HTML markup as possible, particularly content not associated with any entry (e.g. introduction, changelog, etc.).
 * Names must be as short as possible and unique across the documentation.
 * The number of types (categories) should ideally be less than 100.
-* Don't modify the icon sprite. I'll do it after your pull request is merged.
 ## Updating existing documentations
-Please don't submit a pull request updating the version number of a documentation, unless a change is required in the scraper and you've verified that it works.
+Please don't submit a pull request updating only the version number and/or the attribution of a documentation. Do submit a pull request if a bigger change is required in the scraper and you've verified that it works.
-To ask that an existing documentation be updated, please use the [Trello board](https://trello.com/c/2B0hmW7M/52-request-updates-here).
-## Other contributions
-Besides new docs and features, here are other ways you can contribute:
-* **Improve our copy.** English isn't my first language so if you notice grammatical or usage errors, feel free to submit a pull request — it'll be much appreciated.
+To ask that an existing documentation be updated, first check the latest [documentation versions report](https://github.com/freeCodeCamp/devdocs/issues?utf8=%E2%9C%93&q=Documentation+versions+report+is%3Aissue+author%3Adevdocs-bot+sort%3Acreated-desc). Only create an issue if the documentation has been wrongly marked as up-to-date.
-* **Participate in the issue tracker.** Your opinion matters — feel free to add comments to existing issues. You're also welcome to participate to the [mailing list](https://groups.google.com/d/forum/devdocs).
 ## Coding conventions
 ## Questions?
-If you have any questions, please feel free to ask on the [mailing list](https://groups.google.com/d/forum/devdocs).
+If you have any questions, please feel free to ask them on the contributor chat room on [Gitter](https://gitter.im/FreeCodeCamp/DevDocs).

.github/ISSUE_TEMPLATE.md+2−2

@@ -1,7 +1,7 @@
1	<!--	1	<!--
2	Please read the contributing guidelines before opening an issue:	2	Please read the contributing guidelines before opening an issue:
3	https://github.com/freeCodeCamp/devdocs/blob/master/CONTRIBUTING.md	3	https://github.com/freeCodeCamp/devdocs/blob/master/.github/CONTRIBUTING.md
4		4
5	To request a new documentation, or an update of an existing documentation, go here:	5	Go here to request a new documentation:
6	https://trello.com/b/6BmTulfx/devdocs-documentation	6	https://trello.com/b/6BmTulfx/devdocs-documentation
7	-->	7	-->

.gitignore+3−4

@@ -1,10 +1,9 @@
1	.DS_Store	1	.DS_Store
2	.bundle	2	.bundle
3	*.pxm	3	log
4	*.sketch
5	tmp	4	tmp
6	public/assets	5	public/assets
7	public/fonts	6	public/fonts
8	public/docs/*/	7	public/docs/*/
9	!public/docs/docs.json	8	docs/*/
10	!public/docs/**/index.json	9	!docs/*.md

.ruby-version+1−1

@@ -1 +1 @@
1	2.5.1	1	2.6.3

.slugignore+1−2

@@ -1,2 +1 @@
1	public/icons	1	test
2	test

.travis.yml+20−1

@@ -1,7 +1,26 @@
1	language: ruby	1	language: ruby
2		2
		3	addons:
		4	apt:
		5	packages:
		6	- libcurl4-openssl-dev
		7
3	cache: bundler	8	cache: bundler
4		9
5	before_script:	10	before_install:
		11	- "echo 'gem: --no-document' > ~/.gemrc"
6	- gem update --system	12	- gem update --system
7	- gem install bundler	13	- gem install bundler
		14
		15	script:
		16	- if [ "$TRAVIS_EVENT_TYPE" != "cron" ]; then bundle exec rake; fi
		17	- if [ "$TRAVIS_EVENT_TYPE" = "cron" ]; then bundle exec thor updates:check --github-token $GH_TOKEN --upload; fi
		18
		19	deploy:
		20	provider: heroku
		21	app: devdocs
		22	on:
		23	branch: master
		24	condition: $TRAVIS_EVENT_TYPE != cron
		25	api_key:
		26	secure: "4p1klvWJZSOImzFcKOduILjP93hlOlAhceWlYMKS4tU+TCFE8qTBzdKdFPSCsCgjB+YR9pBss+L0lJpVVMjSwFHXqpKe6EeUSltO2k7DFHfW7kXLUM/L0AfqXz+YXk76XUyZMhvOEbldPfaMaj10e8vgDOQCSHABDyK/4CU+hnI="

CODE_OF_CONDUCT.md+2−0

@@ -0,0 +1,2 @@
	1
	2	> Our Code of Conduct is available here: <https://code-of-conduct.freecodecamp.org/>

COPYRIGHT+4−4

@@ -1,13 +1,13 @@
1	Copyright 2013-2018 Thibaut Courouble and other contributors	1	Copyright 2013-2019 Thibaut Courouble and other contributors
2		2
3	This Source Code Form is subject to the terms of the Mozilla Public	3	This Source Code Form is subject to the terms of the Mozilla Public
4	License, v. 2.0. If a copy of the MPL was not distributed with this	4	License, v. 2.0. If a copy of the MPL was not distributed with this
5	file, You can obtain one at http://mozilla.org/MPL/2.0/.	5	file, You can obtain one at http://mozilla.org/MPL/2.0/.
6		6
7	Please do not use the name DevDocs to endorse or promote products	7	Please do not use the name DevDocs to endorse or promote products
8	derived from this software without my permission, except as may be	8	derived from this software without the maintainers' permission, except
9	necessary to comply with the notice/attribution requirements.	9	as may be necessary to comply with the notice/attribution requirements.
10		10
11	I also wish that any documentation file generated using this software	11	We also wish that any documentation file generated using this software
12	be attributed to DevDocs. Let's be fair to all contributors by giving	12	be attributed to DevDocs. Let's be fair to all contributors by giving
13	credit where credit's due. Thanks.	13	credit where credit's due. Thanks.

Dockerfile+3−2

-FROM ruby:2.5.1
+FROM ruby:2.6.3
 ENV LANG=C.UTF-8
+ENV ENABLE_SERVICE_WORKER=true
 WORKDIR /devdocs
 RUN apt-get update && \
-    apt-get -y install git nodejs && \
+    apt-get -y install git nodejs libcurl4 && \
     gem install bundler && \
     rm -rf /var/lib/apt/lists/*

Dockerfile-alpine+3−2

-FROM ruby:2.5.1-alpine
+FROM ruby:2.6.3-alpine
 ENV LANG=C.UTF-8
+ENV ENABLE_SERVICE_WORKER=true
 WORKDIR /devdocs
 COPY . /devdocs
-RUN apk --update add nodejs build-base libstdc++ gzip git zlib-dev && \
+RUN apk --update add nodejs build-base libstdc++ gzip git zlib-dev libcurl && \
     gem install bundler && \
     bundle install --system --without test && \
     thor docs:download --all && \

Gemfile+6−2

 source 'https://rubygems.org'
-ruby '2.5.1'
+ruby '2.6.3'
 gem 'rake'
 gem 'thor'
-gem 'pry', '~> 0.11.0'
+gem 'pry', '~> 0.12.0'
 gem 'activesupport', '~> 5.2', require: false
 gem 'yajl-ruby', require: false
 gem 'html-pipeline'
   gem 'browser'
   gem 'sass'
   gem 'coffee-script'
+  gem 'chunky_png'
+  gem 'sprockets-sass'
 end
 group :production do
   gem 'progress_bar', require: false
   gem 'unix_utils', require: false
   gem 'tty-pager', require: false
+  gem 'net-sftp', '>= 2.1.3.rc2', require: false
+  gem 'terminal-table', require: false
 end
 group :test do

Gemfile.lock+66−55

 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (5.2.1)
+    activesupport (5.2.3)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 0.7, < 2)
       minitest (~> 5.1)
       tzinfo (~> 1.1)
-    backports (3.11.4)
+    backports (3.15.0)
-    better_errors (2.5.0)
+    better_errors (2.5.1)
       coderay (>= 1.0.0)
       erubi (>= 1.0.0)
       rack (>= 0.9.0)
-    browser (2.5.3)
+    browser (2.6.1)
+    chunky_png (1.3.11)
     coderay (1.1.2)
     coffee-script (2.4.1)
       coffee-script-source
       execjs
     coffee-script-source (1.12.2)
-    concurrent-ruby (1.0.5)
+    concurrent-ruby (1.1.5)
-    daemons (1.2.6)
+    daemons (1.3.1)
-    erubi (1.7.1)
+    erubi (1.8.0)
-    ethon (0.11.0)
+    ethon (0.12.0)
       ffi (>= 1.3.0)
     eventmachine (1.2.7)
     execjs (2.7.0)
-    exifr (1.3.4)
+    exifr (1.3.6)
-    ffi (1.9.25)
+    ffi (1.11.1)
-    fspath (3.1.0)
+    fspath (3.1.2)
-    highline (2.0.0)
+    highline (2.0.2)
-    html-pipeline (2.8.4)
+    html-pipeline (2.11.1)
       activesupport (>= 2)
       nokogiri (>= 1.4)
-    i18n (1.1.0)
+    i18n (1.6.0)
       concurrent-ruby (~> 1.0)
-    image_optim (0.26.2)
+    image_optim (0.26.5)
       exifr (~> 1.2, >= 1.2.2)
       fspath (~> 3.0)
       image_size (>= 1.5, < 3)
       in_threads (~> 1.3)
       progress (~> 3.0, >= 3.0.1)
-    image_optim_pack (0.5.1)
+    image_optim_pack (0.5.6)
       fspath (>= 2.1, < 4)
       image_optim (~> 0.19)
-    image_size (2.0.0)
+    image_size (2.0.2)
-    in_threads (1.5.0)
+    in_threads (1.5.3)
-    method_source (0.9.0)
+    method_source (0.9.2)
-    mini_portile2 (2.3.0)
+    mini_portile2 (2.4.0)
     minitest (5.11.3)
     multi_json (1.13.1)
     mustermann (1.0.3)
-    newrelic_rpm (5.4.0.347)
+    net-sftp (3.0.0.beta1)
-    nokogiri (1.8.4)
+      net-ssh (>= 5.0.0, < 6.0.0)
-      mini_portile2 (~> 2.3.0)
+    net-ssh (5.2.0)
+    newrelic_rpm (6.5.0.357)
+    nokogiri (1.10.3)
+      mini_portile2 (~> 2.4.0)
     options (2.3.2)
-    progress (3.4.0)
+    progress (3.5.2)
     progress_bar (1.3.0)
       highline (>= 1.6, < 3)
       options (~> 2.3.0)
-    pry (0.11.3)
+    pry (0.12.2)
       coderay (~> 1.1.0)
       method_source (~> 0.9.0)
-    rack (2.0.5)
+    rack (2.0.7)
-    rack-protection (2.0.4)
+    rack-protection (2.0.5)
       rack
     rack-ssl-enforcer (0.2.9)
     rack-test (1.1.0)
       rack (>= 1.0, < 3)
-    rake (12.3.1)
+    rake (12.3.2)
     rb-fsevent (0.10.3)
-    rb-inotify (0.9.10)
+    rb-inotify (0.10.0)
-      ffi (>= 0.5.0, < 2)
+      ffi (~> 1.0)
     rr (1.2.1)
-    sass (3.6.0)
+    sass (3.7.4)
       sass-listen (~> 4.0.0)
     sass-listen (4.0.0)
       rb-fsevent (~> 0.9, >= 0.9.4)
       rb-inotify (~> 0.9, >= 0.9.7)
-    sinatra (2.0.4)
+    sinatra (2.0.5)
       mustermann (~> 1.0)
       rack (~> 2.0)
-      rack-protection (= 2.0.4)
+      rack-protection (= 2.0.5)
       tilt (~> 2.0)
-    sinatra-contrib (2.0.4)
+    sinatra-contrib (2.0.5)
-      activesupport (>= 4.0.0)
       backports (>= 2.8.2)
       multi_json
       mustermann (~> 1.0)
-      rack-protection (= 2.0.4)
+      rack-protection (= 2.0.5)
-      sinatra (= 2.0.4)
+      sinatra (= 2.0.5)
       tilt (>= 1.3, < 3)
     sprockets (3.7.2)
       concurrent-ruby (~> 1.0)
       rack (> 1, < 3)
     sprockets-helpers (1.2.1)
       sprockets (>= 2.2)
-    strings (0.1.4)
+    sprockets-sass (2.0.0.beta2)
-      strings-ansi (~> 0.1.0)
+      sprockets (>= 2.0, < 4.0)
-      unicode-display_width (~> 1.4.0)
+    strings (0.1.5)
-      unicode_utils (~> 1.4.0)
+      strings-ansi (~> 0.1)
+      unicode-display_width (~> 1.5)
+      unicode_utils (~> 1.4)
     strings-ansi (0.1.0)
+    terminal-table (1.8.0)
+      unicode-display_width (~> 1.1, >= 1.1.1)
     thin (1.7.2)
       daemons (~> 1.0, >= 1.0.9)
       eventmachine (~> 1.0, >= 1.0.4)
       rack (>= 1, < 3)
-    thor (0.20.0)
+    thor (0.20.3)
     thread_safe (0.3.6)
-    tilt (2.0.8)
+    tilt (2.0.9)
-    tty-pager (0.11.0)
+    tty-pager (0.12.1)
-      strings (~> 0.1.0)
+      strings (~> 0.1.4)
-      tty-screen (~> 0.6.4)
+      tty-screen (~> 0.6)
-      tty-which (~> 0.3.0)
+      tty-which (~> 0.4)
-    tty-screen (0.6.5)
+    tty-screen (0.7.0)
-    tty-which (0.3.0)
+    tty-which (0.4.1)
-    typhoeus (1.3.0)
+    typhoeus (1.3.1)
       ethon (>= 0.9.0)
     tzinfo (1.2.5)
       thread_safe (~> 0.1)
-    uglifier (4.1.19)
+    uglifier (4.1.20)
       execjs (>= 0.3.0, < 3)
-    unicode-display_width (1.4.0)
+    unicode-display_width (1.6.0)
     unicode_utils (1.4.0)
     unix_utils (0.0.15)
     yajl-ruby (1.4.1)
   activesupport (~> 5.2)
   better_errors
   browser
+  chunky_png
   coffee-script
   erubi
   html-pipeline
   image_optim
   image_optim_pack
   minitest
+  net-sftp (>= 2.1.3.rc2)
   newrelic_rpm
   nokogiri
   progress_bar
-  pry (~> 0.11.0)
+  pry (~> 0.12.0)
   rack
   rack-ssl-enforcer
   rack-test
   sinatra-contrib
   sprockets
   sprockets-helpers
+  sprockets-sass
+  terminal-table
   thin
   thor
   tty-pager
   yajl-ruby
 RUBY VERSION
-   ruby 2.5.1p57
+   ruby 2.6.3p62
 BUNDLED WITH
-   1.16.1
+   2.0.2

PULL_REQUEST_TEMPLATE.md+11−0

@@ -0,0 +1,11 @@
	1	If you’re adding a new scraper, please ensure that you have:
	2
	3	- [ ] Tested the scraper on a local copy of DevDocs
	4	- [ ] Ensured that the docs are styled similarly to other docs on DevDocs
	5	<!-- If the docs don’t have an icon, delete the next four items: -->
	6	- [ ] Added these files to the <code>public/icons/your_scraper_name/</code> directory:
	7	- [ ] `16.png`: a 16×16 pixel icon for the doc
	8	- [ ] `16@2x.png`: a 32×32 pixel icon for the doc
	9	- [ ] `SOURCE`: A text file containing the URL to the page the image can be found on or the URL of the original image itself
	10
	11	<!-- Replace the `[ ]` with a `[x]` once you’ve completed each step. -->

README.md+45−20

-# [DevDocs](https://devdocs.io) [![Build Status](https://travis-ci.org/freeCodeCamp/devdocs.svg?branch=master)](https://travis-ci.org/freeCodeCamp/devdocs)
+# [DevDocs](https://devdocs.io) — API Documentation Browser [![Build Status](https://travis-ci.org/freeCodeCamp/devdocs.svg?branch=master)](https://travis-ci.org/freeCodeCamp/devdocs)
-DevDocs combines multiple API documentations in a fast, organized, and searchable interface.
+DevDocs combines multiple developer documentations in a clean and organized web UI with instant search, offline support, mobile version, dark theme, keyboard shortcuts, and more.
-* Created by [Thibaut Courouble](https://thibaut.me)
+DevDocs was created by [Thibaut Courouble](https://thibaut.me) and is operated by [freeCodeCamp](https://www.freecodecamp.org).
 Keep track of development news:
 * Join the contributor chat room on [Gitter](https://gitter.im/FreeCodeCamp/DevDocs)
 * Watch the repository on [GitHub](https://github.com/freeCodeCamp/devdocs/subscription)
 * Follow [@DevDocs](https://twitter.com/DevDocs) on Twitter
-* Join the [mailing list](https://groups.google.com/d/forum/devdocs)
-**Table of Contents:** [Quick Start](#quick-start) · [Vision](#vision) · [App](#app) · [Scraper](#scraper) · [Commands](#available-commands) · [Contributing](#contributing) · [License](#copyright--license) · [Questions?](#questions)
+**Table of Contents:** [Quick Start](#quick-start) · [Vision](#vision) · [App](#app) · [Scraper](#scraper) · [Commands](#available-commands) · [Contributing](#contributing) · [Documentation](#documentation) · [Plugins and Extensions](#plugins-and-extensions) · [License](#copyright--license) · [Questions?](#questions)
 ## Quick Start
-Unless you wish to contribute to the project, I recommend using the hosted version at [devdocs.io](https://devdocs.io). It's up-to-date and works offline out-of-the-box.
+Unless you wish to contribute to the project, we recommend using the hosted version at [devdocs.io](https://devdocs.io). It's up-to-date and works offline out-of-the-box.
 DevDocs is made of two pieces: a Ruby scraper that generates the documentation and metadata, and a JavaScript app powered by a small Sinatra app.
-DevDocs requires Ruby 2.5.1, libcurl, and a JavaScript runtime supported by [ExecJS](https://github.com/rails/execjs#readme) (included in OS X and Windows; [Node.js](https://nodejs.org/en/) on Linux). Once you have these installed, run the following commands:
+DevDocs requires Ruby 2.6.0, libcurl, and a JavaScript runtime supported by [ExecJS](https://github.com/rails/execjs#readme) (included in OS X and Windows; [Node.js](https://nodejs.org/en/) on Linux). Once you have these installed, run the following commands:
 ```
 git clone https://github.com/freeCodeCamp/devdocs.git && cd devdocs
 Many of the code's design decisions were driven by the fact that the app uses XHR to load content directly into the main frame. This includes stripping the original documents of most of their HTML markup (e.g. scripts and stylesheets) to avoid polluting the main frame, and prefixing all CSS class names with an underscore to prevent conflicts.
-Another driving factor is performance and the fact that everything happens in the browser. `applicationCache` (which comes with its own set of constraints) and `localStorage` are used to speed up the boot time, while memory consumption is kept in check by allowing the user to pick his/her own set of documentations. The search algorithm is kept simple because it needs to be fast even searching through 100,000 strings.
+Another driving factor is performance and the fact that everything happens in the browser. A service worker (which comes with its own set of constraints) and `localStorage` are used to speed up the boot time, while memory consumption is kept in check by allowing the user to pick his/her own set of documentations. The search algorithm is kept simple because it needs to be fast even searching through 100,000 strings.
 DevDocs being a developer tool, the browser requirements are high:
 * Recent versions of Firefox, Chrome, or Opera
-* Safari 9.1+
+* Safari 11.1+
-* Edge 16+
+* Edge 17+
-* iOS 10+
+* iOS 11.3+
 This allows the code to take advantage of the latest DOM and HTML5 APIs and make developing DevDocs a lot more fun!
 * replacing all external (not scraped) URLs with their fully qualified counterpart
 * replacing all internal (scraped) URLs with their unqualified and relative counterpart
 * adding content, such as a title and link to the original document
+* ensuring correct syntax highlighting using [Prism](http://prismjs.com/)
 These modifications are applied via a set of filters using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. Each scraper includes filters specific to itself, one of which is tasked with figuring out the pages' metadata.
 The end result is a set of normalized HTML partials and two JSON files (index + offline data). Because the index files are loaded separately by the [app](#app) following the user's preferences, the scraper also creates a JSON manifest file containing information about the documentations currently available on the system (such as their name, version, update date, etc.).
-More information about scrapers and filters is available on the [wiki](https://github.com/freeCodeCamp/devdocs/wiki).
+More information about [scrapers](./docs/scraper-reference.md) and [filters](./docs/filter-reference.md) is available in the `docs` folder.
 ## Available Commands
 ## Contributing
-Contributions are welcome. Please read the [contributing guidelines](https://github.com/freeCodeCamp/devdocs/blob/master/CONTRIBUTING.md).
+Contributions are welcome. Please read the [contributing guidelines](./.github/CONTRIBUTING.md).
-DevDocs's own documentation is available on the [wiki](https://github.com/freeCodeCamp/devdocs/wiki).
+## Documentation
+* [Adding documentations to DevDocs](./docs/adding-docs.md)
+* [Scraper Reference](./docs/scraper-reference.md)
+* [Filter Reference](./docs/filter-reference.md)
+* [Maintainers’ Guide](./docs/maintainers.md)
+## Plugins and Extensions
+* [Chrome web app](https://chrome.google.com/webstore/detail/devdocs/mnfehgbmkapmjnhcnbodoamcioleeooe)
+* [Ubuntu Touch app](https://uappexplorer.com/app/devdocsunofficial.berkes)
+* [Sublime Text plugin](https://sublime.wbond.net/packages/DevDocs)
+* [Atom plugin](https://atom.io/packages/devdocs)
+* [Brackets extension](https://github.com/gruehle/dev-docs-viewer)
+* [Fluid](http://fluidapp.com) for turning DevDocs into a real OS X app
+* [GTK shell / Vim integration](https://github.com/naquad/devdocs-shell)
+* [Emacs lookup](https://github.com/skeeto/devdocs-lookup)
+* [Alfred Workflow](https://github.com/yannickglt/alfred-devdocs)
+* [Vim search plugin with Devdocs in its defaults](https://github.com/waiting-for-dev/vim-www) Just set `let g:www_shortcut_engines = { 'devdocs': ['Devdocs', '<leader>dd'] }` to have a `:Devdocs` command and a `<leader>dd` mapping.
+* [Visual Studio Code plugin](https://marketplace.visualstudio.com/items?itemName=akfish.vscode-devdocs ) (1)
+* [Visual Studio Code plugin](https://marketplace.visualstudio.com/items?itemName=deibit.devdocs) (2)
+* [Desktop application](https://github.com/egoist/devdocs-desktop)
+* [Doc Browser](https://github.com/qwfy/doc-browser) is a native Linux app that supports DevDocs docsets
+* [GNOME Application](https://github.com/hardpixel/devdocs-desktop) GTK3 application with search integrated in headerbar
+* [macOS Application](https://github.com/dteoh/devdocs-macos)
+* [Android Application](https://github.com/Merith-TK/devdocs_webapp_kotlin) is a fully working, advanced WebView
 ## Copyright / License
-Copyright 2013-2018 Thibaut Courouble and [other contributors](https://github.com/freeCodeCamp/devdocs/graphs/contributors)
+Copyright 2013-2019 Thibaut Courouble and [other contributors](https://github.com/freeCodeCamp/devdocs/graphs/contributors)
-This software is licensed under the terms of the Mozilla Public License v2.0. See the [COPYRIGHT](https://github.com/freeCodeCamp/devdocs/blob/master/COPYRIGHT) and [LICENSE](https://github.com/freeCodeCamp/devdocs/blob/master/LICENSE) files.
+This software is licensed under the terms of the Mozilla Public License v2.0. See the [COPYRIGHT](./COPYRIGHT) and [LICENSE](./LICENSE) files.
-Please do not use the name DevDocs to endorse or promote products derived from this software without my permission, except as may be necessary to comply with the notice/attribution requirements.
+Please do not use the name DevDocs to endorse or promote products derived from this software without the maintainers' permission, except as may be necessary to comply with the notice/attribution requirements.
-I also wish that any documentation file generated using this software be attributed to DevDocs. Let's be fair to all contributors by giving credit where credit's due. Thanks!
+We also wish that any documentation file generated using this software be attributed to DevDocs. Let's be fair to all contributors by giving credit where credit's due. Thanks!
 ## Questions?
-If you have any questions, please feel free to ask them on the [mailing list](https://groups.google.com/d/forum/devdocs).
+If you have any questions, please feel free to ask them on the contributor chat room on [Gitter](https://gitter.im/FreeCodeCamp/DevDocs).

assets/images/.gitignore+1−0

@@ -0,0 +1 @@
		1	sprites/*/

assets/images/docs-1.png+0−0

Binary file not shown.

assets/images/docs-1@2x.png+0−0

Binary file not shown.

assets/images/docs-2.png+0−0

Binary file not shown.

assets/images/docs-2@2x.png+0−0

Binary file not shown.

assets/javascripts/app/app.coffee+15−10

     @el = $('._app')
     @localStorage = new LocalStorageStore
-    @appCache = new app.AppCache if app.AppCache.isEnabled()
+    @serviceWorker = new app.ServiceWorker if app.ServiceWorker.isEnabled()
     @settings = new app.Settings
     @db = new app.DB()
+    @settings.initLayout()
     @docs = new app.collections.Docs
     @disabledDocs = new app.collections.Docs
     @entries = new app.collections.Entries
         .install()
       @previousErrorHandler = onerror
       window.onerror = @onWindowError.bind(@)
-      CookieStore.onBlocked = @onCookieBlocked
+      CookiesStore.onBlocked = @onCookieBlocked
     return
   bootOne: ->
       @docs.sort()
       @initDoc(doc)
       @saveDocs()
-      _onSuccess()
+      if app.settings.get('autoInstall')
+        doc.install(_onSuccess, onError)
+      else
+        _onSuccess()
       return
     doc.load onSuccess, onError, writeCache: true
   saveDocs: ->
     @settings.setDocs(doc.slug for doc in @docs.all())
     @db.migrate()
-    @appCache?.updateInBackground()
+    @serviceWorker?.updateInBackground()
   welcomeBack: ->
     visitCount = @settings.get('count')
     @updateChecker = new app.UpdateChecker()
   reboot: ->
-    if location.pathname isnt '/'
+    if location.pathname isnt '/' and location.pathname isnt '/settings'
       window.location = "/##{location.pathname}"
     else
       window.location = '/'
   reload: ->
     @docs.clearCache()
     @disabledDocs.clearCache()
-    if @appCache then @appCache.reload() else @reboot()
+    if @serviceWorker then @serviceWorker.reload() else @reboot()
     return
   reset: ->
     @localStorage.reset()
     @settings.reset()
     @db?.reset()
-    @appCache?.update()
+    @serviceWorker?.update()
     window.location = '/'
     return
     return
   indexHost: ->
-    # Can't load the index files from the host/CDN when applicationCache is
+    # Can't load the index files from the host/CDN when service worker is
     # enabled because it doesn't support caching URLs that use CORS.
-    @config[if @appCache and @settings.hasDocs() then 'index_path' else 'docs_origin']
+    @config[if @serviceWorker and @settings.hasDocs() then 'index_path' else 'docs_origin']
   onBootError: (args...) ->
     @trigger 'bootError'
         matchMedia:         !!window.matchMedia
         insertAdjacentHTML: !!document.body.insertAdjacentHTML
         defaultPrevented:     document.createEvent('CustomEvent').defaultPrevented is false
-        cssVariables:         CSS.supports and CSS.supports('(--t: 0)')
+        cssVariables:       !!CSS?.supports?('(--t: 0)')
       for key, value of features when not value
         Raven.captureMessage "unsupported/#{key}", level: 'info'

assets/javascripts/app/appcache.coffee+0−46

@@ -1,46 +0,0 @@
1	class app.AppCache
2	$.extend @prototype, Events
3
4	@isEnabled: ->
5	try
6	applicationCache and applicationCache.status isnt applicationCache.UNCACHED
7	catch
8
9	constructor: ->
10	@cache = applicationCache
11	@notifyUpdate = true
12	@onUpdateReady() if @cache.status is @cache.UPDATEREADY
13
14	$.on @cache, 'progress', @onProgress
15	$.on @cache, 'updateready', @onUpdateReady
16
17	update: ->
18	@notifyUpdate = true
19	@notifyProgress = true
20	try @cache.update() catch
21	return
22
23	updateInBackground: ->
24	@notifyUpdate = false
25	@notifyProgress = false
26	try @cache.update() catch
27	return
28
29	reload: ->
30	$.on @cache, 'updateready noupdate error', ->
31	if location.pathname isnt '/'
32	window.location = "/##{location.pathname}"
33	else
34	window.location = '/'
35	@notifyUpdate = false
36	@notifyProgress = true
37	try @cache.update() catch
38	return
39
40	onProgress: (event) =>
41	@trigger 'progress', event if @notifyProgress
42	return
43
44	onUpdateReady: =>
45	@trigger 'updateready' if @notifyUpdate
46	return

assets/javascripts/app/config.coffee.erb+3−0

@@ -13,3 +13,6 @@ app.config = · 12 unmodified lines
13	version: <%= Time.now.to_i %>	13	version: <%= Time.now.to_i %>
14	release: <%= Time.now.utc.httpdate.to_json %>	14	release: <%= Time.now.utc.httpdate.to_json %>
15	mathml_stylesheet: '<%= App.cdn_origin %>/mathml.css'	15	mathml_stylesheet: '<%= App.cdn_origin %>/mathml.css'
		16	favicon_spritesheet: '<%= image_path('sprites/docs.png') %>'
		17	service_worker_path: '/service-worker.js'
		18	service_worker_enabled: <%= App.environment == :production \|\| ENV['ENABLE_SERVICE_WORKER'] == 'true' %>

assets/javascripts/app/serviceworker.coffee+49−0

@@ -0,0 +1,49 @@
	1	class app.ServiceWorker
	2	$.extend @prototype, Events
	3
	4	@isEnabled: ->
	5	!!navigator.serviceWorker and app.config.service_worker_enabled
	6
	7	constructor: ->
	8	@registration = null
	9	@notifyUpdate = true
	10
	11	navigator.serviceWorker.register(app.config.service_worker_path, {scope: '/'})
	12	.then(
	13	(registration) => @updateRegistration(registration),
	14	(error) -> console.error('Could not register service worker:', error)
	15	)
	16
	17	update: ->
	18	return unless @registration
	19	@notifyUpdate = true
	20	return @registration.update().catch(->)
	21
	22	updateInBackground: ->
	23	return unless @registration
	24	@notifyUpdate = false
	25	return @registration.update().catch(->)
	26
	27	reload: ->
	28	return @updateInBackground().then(() -> app.reboot())
	29
	30	updateRegistration: (registration) ->
	31	@registration = registration
	32	$.on @registration, 'updatefound', @onUpdateFound
	33	return
	34
	35	onUpdateFound: =>
	36	$.off @installingRegistration, 'statechange', @onStateChange() if @installingRegistration
	37	@installingRegistration = @registration.installing
	38	$.on @installingRegistration, 'statechange', @onStateChange
	39	return
	40
	41	onStateChange: =>
	42	if @installingRegistration and @installingRegistration.state == 'installed' and navigator.serviceWorker.controller
	43	@installingRegistration = null
	44	@onUpdateReady()
	45	return
	46
	47	onUpdateReady: ->
	48	@trigger 'updateready' if @notifyUpdate
	49	return

assets/javascripts/app/settings.coffee+34−1

     'manualUpdate'
     'fastScroll'
     'arrowScroll'
+    'analyticsConsent'
     'docs'
     'dark'
     'layout'
     'size'
     'tips'
+    'autoInstall'
   ]
   INTERNAL_KEYS = [
     'news'
   ]
+  LAYOUTS: ['_max-width', '_sidebar-hidden', '_native-scrollbars']
   @defaults:
     count: 0
     hideDisabled: false
     news: 0
     manualUpdate: false
     schema: 1
+    analyticsConsent: false
   constructor: ->
-    @store = new CookieStore
+    @store = new CookiesStore
     @cache = {}
   get: (key) ->
   set: (key, value) ->
     @store.set(key, value)
     delete @cache[key]
+    @toggleDark(value) if key == 'dark'
     return
   del: (key) ->
     return
   setLayout: (name, enable) ->
+    @toggleLayout(name, enable)
     layout = (@store.get('layout') || '').split(' ')
     $.arrayDelete(layout, '')
     @store.reset()
     @cache = {}
     return
+  initLayout: ->
+    @toggleDark(@get('dark') is 1)
+    @toggleLayout(layout, @hasLayout(layout)) for layout in @LAYOUTS
+    @initSidebarWidth()
+    return
+  toggleDark: (enable) ->
+    classList = document.documentElement.classList
+    classList.toggle('_theme-default', !enable)
+    classList.toggle('_theme-dark', enable)
+    color = getComputedStyle(document.documentElement).getPropertyValue('--headerBackground').trim()
+    $('meta[name=theme-color]').setAttribute('content', color)
+    return
+  toggleLayout: (layout, enable) ->
+    classList = document.body.classList
+    classList.toggle(layout, enable) unless layout is '_sidebar-hidden'
+    classList.toggle('_overlay-scrollbars', $.overlayScrollbarsEnabled())
+    return
+  initSidebarWidth: ->
+    size = @get('size')
+    document.documentElement.style.setProperty('--sidebarWidth', size + 'px') if size
+    return

assets/javascripts/app/update_checker.coffee+3−3

@@ -3,13 +3,13 @@ class app.UpdateChecker · 2 unmodified lines
3	@lastCheck = Date.now()	3	@lastCheck = Date.now()
4		4
5	$.on window, 'focus', @onFocus	5	$.on window, 'focus', @onFocus
6	app.appCache.on 'updateready', @onUpdateReady if app.appCache	6	app.serviceWorker?.on 'updateready', @onUpdateReady
7		7
8	setTimeout @checkDocs, 0	8	setTimeout @checkDocs, 0
9		9
10	check: ->	10	check: ->
11	if app.appCache	11	if app.serviceWorker
12	app.appCache.update()	12	app.serviceWorker.update()
13	else	13	else
14	ajax	14	ajax
15	url: $('script[src*="application"]').getAttribute('src')	15	url: $('script[src*="application"]').getAttribute('src')

assets/javascripts/lib/cookies_store.coffeerenamed from assets/javascripts/lib/cookie_store.coffee+5−1

@@ -1,4 +1,8 @@
1	class @CookieStore	1	class @CookiesStore
		2	# Intentionally called CookiesStore instead of CookieStore
		3	# Calling it CookieStore causes issues when the Experimental Web Platform features flag is enabled in Chrome
		4	# Related issue: https://github.com/freeCodeCamp/devdocs/issues/932
		5
2	INT = /^\d+$/	6	INT = /^\d+$/
3		7
4	@onBlocked: ->	8	@onBlocked: ->

assets/javascripts/lib/favicon.coffee+64−0

@@ -0,0 +1,64 @@
	1	defaultUrl = null
	2	currentSlug = null
	3
	4	imageCache = {}
	5	urlCache = {}
	6
	7	withImage = (url, action) ->
	8	if imageCache[url]
	9	action(imageCache[url])
	10	else
	11	img = new Image()
	12	img.crossOrigin = 'anonymous'
	13	img.src = url
	14	img.onload = () =>
	15	imageCache[url] = img
	16	action(img)
	17
	18	@setFaviconForDoc = (doc) ->
	19	return if currentSlug == doc.slug
	20
	21	favicon = $('link[rel="icon"]')
	22
	23	if defaultUrl == null
	24	defaultUrl = favicon.href
	25
	26	if urlCache[doc.slug]
	27	favicon.href = urlCache[doc.slug]
	28	currentSlug = doc.slug
	29	return
	30
	31	styles = window.getComputedStyle($("._icon-#{doc.slug.split('~')[0]}"), ':before')
	32
	33	bgUrl = app.config.favicon_spritesheet
	34	sourceSize = 16
	35	sourceX = Math.abs(parseInt(styles['background-position-x'].slice(0, -2)))
	36	sourceY = Math.abs(parseInt(styles['background-position-y'].slice(0, -2)))
	37
	38	withImage(bgUrl, (docImg) ->
	39	withImage(defaultUrl, (defaultImg) ->
	40	size = defaultImg.width
	41
	42	canvas = document.createElement('canvas')
	43	ctx = canvas.getContext('2d')
	44
	45	canvas.width = size
	46	canvas.height = size
	47	ctx.drawImage(defaultImg, 0, 0)
	48
	49	docIconPercentage = 65
	50	destinationCoords = size / 100 * (100 - docIconPercentage)
	51	destinationSize = size / 100 * docIconPercentage
	52	ctx.drawImage(docImg, sourceX, sourceY, sourceSize, sourceSize, destinationCoords, destinationCoords, destinationSize, destinationSize)
	53
	54	urlCache[doc.slug] = canvas.toDataURL()
	55	favicon.href = urlCache[doc.slug]
	56
	57	currentSlug = doc.slug
	58	)
	59	)
	60
	61	@resetFavicon = () ->
	62	if defaultUrl != null and currentSlug != null
	63	$('link[rel="icon"]').href = defaultUrl
	64	currentSlug = null

assets/javascripts/lib/license.coffee+1−1

@@ -1,5 +1,5 @@
1	###	1	###
2	* Copyright 2013-2018 Thibaut Courouble and other contributors	2	* Copyright 2013-2019 Thibaut Courouble and other contributors
3	*	3	*
4	* This source code is licensed under the terms of the Mozilla	4	* This source code is licensed under the terms of the Mozilla
5	* Public License, v. 2.0, a copy of which may be obtained at:	5	* Public License, v. 2.0, a copy of which may be obtained at:

assets/javascripts/lib/page.coffee+17−1

@@ -199,5 +199,21 @@ page.track = (fn) -> · 198 unmodified lines
199	return	199	return
200		200
201	track = ->	201	track = ->
202	tracker.call() for tracker in trackers	202	consentGiven = Cookies.get('analyticsConsent')
		203	consentAsked = Cookies.get('analyticsConsentAsked')
		204
		205	if consentGiven == '1'
		206	tracker.call() for tracker in trackers
		207	else if consentGiven == undefined and consentAsked == undefined
		208	# Only ask for consent once per browser session
		209	Cookies.set('analyticsConsentAsked', '1')
		210
		211	new app.views.Notif 'AnalyticsConsent', autoHide: null
		212	return
		213
		214	@resetAnalytics = ->
		215	for cookie in document.cookie.split(/;\s?/)
		216	name = cookie.split('=')[0]
		217	if name[0] == '_' && name[1] != '_'
		218	Cookies.expire(name)
203	return	219	return

assets/javascripts/lib/util.coffee+4−0

 $.isIE = ->
   isIE ?= navigator.userAgent?.indexOf('MSIE') >= 0 || navigator.userAgent?.indexOf('rv:11.0') >= 0
+isChromeForAndroid = null
+$.isChromeForAndroid = ->
+  isChromeForAndroid ?= navigator.userAgent?.indexOf('Android') >= 0 && /Chrome\/([.0-9])+ Mobile/.test(navigator.userAgent)
 isAndroid = null
 $.isAndroid = ->
   isAndroid ?= navigator.userAgent?.indexOf('Android') >= 0

assets/javascripts/models/doc.coffee+3−1

@@ -142,4 +142,6 @@ class app.models.Doc extends app.Model · 141 unmodified lines
142	return	142	return
143		143
144	isOutdated: (status) ->	144	isOutdated: (status) ->
145	status and status.installed and @mtime isnt status.mtime	145	return false if not status
		146	isInstalled = status.installed or app.settings.get('autoInstall')
		147	isInstalled and @mtime isnt status.mtime

assets/javascripts/news.json+8−0

@@ -1,4 +1,12 @@
1	[	1	[
		2	[
		3	"2019-07-21",
		4	"Fixed several bugs, added an option to automatically download documentation and <a href=\"https://github.com/freeCodeCamp/devdocs/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+sort%3Aupdated-desc+closed%3A%3E2019-07-18+977+988+986+870+886+1024+979+975+941+831+1005+848+942\" target=\"_blank\">more</a>."
		5	],
		6	[
		7	"2019-07-19",
		8	"Replaced the AppCache with a Service Worker (which makes DevDocs an installable PWA) and fixed layout preferences on Firefox."
		9	],
2	[	10	[
3	"2018-09-23",	11	"2018-09-23",
4	"New documentations: <a href=\"/puppeteer/\">Puppeteer</a> and <a href=\"/handlebars/\">Handlebars.js</a>"	12	"New documentations: <a href=\"/puppeteer/\">Puppeteer</a> and <a href=\"/handlebars/\">Handlebars.js</a>"

assets/javascripts/templates/error_tmpl.coffee+8−8

@@ -12,8 +12,8 @@ app.templates.notFoundPage = -> · 11 unmodified lines
12		12
13	app.templates.pageLoadError = ->	13	app.templates.pageLoadError = ->
14	error """ The page failed to load. """,	14	error """ The page failed to load. """,
15	""" It may be missing from the server (try reloading the app) or you could be offline.<br>	15	""" It may be missing from the server (try reloading the app) or you could be offline (try <a href="/offline">installing the documentation for offline usage</a> when online again).<br>
16	If you keep seeing this, you're likely behind a proxy or firewall that blocks cross-domain requests. """,	16	If you're online and you keep seeing this, you're likely behind a proxy or firewall that blocks cross-domain requests. """,
17	""" #{back} · <a href="/##{location.pathname}" target="_top" class="_error-link">Reload</a>	17	""" #{back} · <a href="/##{location.pathname}" target="_top" class="_error-link">Reload</a>
18	· <a href="#" class="_error-link" data-retry>Retry</a> """	18	· <a href="#" class="_error-link" data-retry>Retry</a> """
19		19
@@ -57,17 +57,17 @@ app.templates.unsupportedBrowser = """ · 37 unmodified lines
57	<p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:	57	<p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:
58	<ul class="_fail-list">	58	<ul class="_fail-list">
59	<li>Recent versions of Firefox, Chrome, or Opera	59	<li>Recent versions of Firefox, Chrome, or Opera
60	<li>Safari 9.1+	60	<li>Safari 11.1+
61	<li>Edge 16+	61	<li>Edge 17+
62	<li>iOS 10+	62	<li>iOS 11.3+
63	</ul>	63	</ul>
64	<p class="_fail-text">	64	<p class="_fail-text">
65	If you're unable to upgrade, I apologize.	65	If you're unable to upgrade, we apologize.
66	I decided to prioritize speed and new features over support for older browsers.	66	We decided to prioritize speed and new features over support for older browsers.
67	<p class="_fail-text">	67	<p class="_fail-text">
68	Note: if you're already using one of the browsers above, check your settings and add-ons.	68	Note: if you're already using one of the browsers above, check your settings and add-ons.
69	The app uses feature detection, not user agent sniffing.	69	The app uses feature detection, not user agent sniffing.
70	<p class="_fail-text">	70	<p class="_fail-text">
71	— Thibaut <a href="https://twitter.com/DevDocs" class="_fail-link">@DevDocs</a>	71	— <a href="https://twitter.com/DevDocs">@DevDocs</a>
72	</div>	72	</div>
73	"""	73	"""

assets/javascripts/templates/notif_tmpl.coffee+6−0

@@ -68,3 +68,9 @@ app.templates.notifShare = -> · 67 unmodified lines
68	app.templates.notifUpdateDocs = ->	68	app.templates.notifUpdateDocs = ->
69	textNotif """ Documentation updates available. """,	69	textNotif """ Documentation updates available. """,
70	""" <a href="/offline">Install them</a> as soon as possible to avoid broken pages. """	70	""" <a href="/offline">Install them</a> as soon as possible to avoid broken pages. """
		71
		72	app.templates.notifAnalyticsConsent = ->
		73	textNotif """ Tracking cookies """,
		74	""" We would like to gather usage data about how DevDocs is used through Google Analytics and Gauges. We only collect anonymous traffic information.
		75	Please confirm if you accept our tracking cookies. You can always change your decision in the settings.
		76	<br><span class="_notif-right"><a href="#" data-behavior="accept-analytics">Accept</a> or <a href="#" data-behavior="decline-analytics">Decline</a></span> """

assets/javascripts/templates/pages/about_tmpl.coffee+54−24

   </nav>
   <h1 class="_lined-heading">DevDocs: API Documentation Browser</h1>
-  <p>DevDocs combines multiple API documentations in a clean and organized web UI with instant search, offline support, mobile version, dark theme, keyboard shortcuts, and more.
+  <p>DevDocs combines multiple developer documentations in a clean and organized web UI with instant search, offline support, mobile version, dark theme, keyboard shortcuts, and more.
-  <ul>
+  <p>DevDocs is free and <a href="https://github.com/freeCodeCamp/devdocs">open source</a>. It was created by <a href="https://thibaut.me">Thibaut Courouble</a> and is operated by <a href="https://www.freecodecamp.org/">freeCodeCamp</a>.
-    <li>Created and maintained by <a href="https://thibaut.me">Thibaut Courouble</a>
-    <li>Free and <a href="https://github.com/freeCodeCamp/devdocs">open source</a>
-        <iframe class="_github-btn" src="https://ghbtns.com/github-btn.html?user=freeCodeCamp&repo=devdocs&type=watch&count=true" allowtransparency="true" frameborder="0" scrolling="0" width="100" height="20" tabindex="-1"></iframe>
-  </ul>
   <p>To keep up-to-date with the latest news:
   <ul>
     <li>Follow <a href="https://twitter.com/DevDocs">@DevDocs</a> on Twitter
-    <li>Watch the repository on <a href="https://github.com/freeCodeCamp/devdocs/subscription">GitHub</a>
+    <li>Watch the repository on <a href="https://github.com/freeCodeCamp/devdocs/subscription">GitHub</a> <iframe class="_github-btn" src="https://ghbtns.com/github-btn.html?user=freeCodeCamp&repo=devdocs&type=watch&count=true" allowtransparency="true" frameborder="0" scrolling="0" width="100" height="20" tabindex="-1"></iframe>
-    <li>Join the <a href="https://groups.google.com/d/forum/devdocs">mailing list</a>
+    <li>Join the <a href="https://gitter.im/FreeCodeCamp/DevDocs">Gitter</a> chat room
   </ul>
   <h2 class="_block-heading" id="copyright">Copyright and License</h2>
   <p class="_note">
-    <strong>Copyright 2013&ndash;2018 Thibaut Courouble and <a href="https://github.com/freeCodeCamp/devdocs/graphs/contributors">other contributors</a></strong><br>
+    <strong>Copyright 2013&ndash;2019 Thibaut Courouble and <a href="https://github.com/freeCodeCamp/devdocs/graphs/contributors">other contributors</a></strong><br>
     This software is licensed under the terms of the Mozilla Public License v2.0.<br>
     You may obtain a copy of the source code at <a href="https://github.com/freeCodeCamp/devdocs">github.com/freeCodeCamp/devdocs</a>.<br>
     For more information, see the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/COPYRIGHT">COPYRIGHT</a>
     <dt>Where can I suggest new docs and features?
     <dd>You can suggest and vote for new docs on the <a href="https://trello.com/b/6BmTulfx/devdocs-documentation">Trello board</a>.<br>
         If you have a specific feature request, add it to the <a href="https://github.com/freeCodeCamp/devdocs/issues">issue tracker</a>.<br>
-        Otherwise use the <a href="https://groups.google.com/d/forum/devdocs">mailing list</a>.
+        Otherwise, come talk to us in the <a href="https://gitter.im/FreeCodeCamp/DevDocs">Gitter</a> chat room.
     <dt>Where can I report bugs?
     <dd>In the <a href="https://github.com/freeCodeCamp/devdocs/issues">issue tracker</a>. Thanks!
   </dl>
-  <p>For anything else, feel free to email me at <a href="mailto:thibaut@devdocs.io">thibaut@devdocs.io</a>.
   <h2 class="_block-heading" id="credits">Credits</h2>
   <h2 class="_block-heading" id="privacy">Privacy Policy</h2>
   <ul>
-    <li><a href="https://devdocs.io">devdocs.io</a> ("App") is operated by Thibaut Courouble ("We").
+    <li><a href="https://devdocs.io">devdocs.io</a> ("App") is operated by <a href="https://www.freecodecamp.org/">freeCodeCamp</a> ("We").
-    <li>We do not collect personal information.
+    <li>We do not collect personal information through the app.
-    <li>We use Google Analytics, Gauges and Sentry to collect anonymous traffic information and improve the app.
+    <li>We use Google Analytics and Gauges to collect anonymous traffic information if you have given consent to this. You can change your decision in the <a href="/settings">settings</a>.
+    <li>We use Sentry to collect crash data and improve the app.
     <li>The app uses cookies to store user preferences.
     <li>By using the app, you signify your acceptance of this policy. If you do not agree to this policy, please do not use the app.
-    <li>If you have any questions regarding privacy, please email <a href="mailto:thibaut@devdocs.io">thibaut@devdocs.io</a>.
+    <li>If you have any questions regarding privacy, please email <a href="mailto:privacy@freecodecamp.org">privacy@freecodecamp.org</a>.
   </ul>
 """
     'https://creativecommons.org/licenses/by/4.0/'
   ], [
     'Ansible',
-    '2012-2018 Michael DeHaan<br>&copy; 2018 Red Hat, Inc.',
+    '2012-2018 Michael DeHaan<br>&copy; 2018–2019 Red Hat, Inc.',
     'GPLv3',
     'https://raw.githubusercontent.com/ansible/ansible/devel/COPYING'
   ], [
     'https://raw.githubusercontent.com/petkaantonov/bluebird/master/LICENSE'
   ], [
     'Bootstrap',
-    '2011-2018 Twitter, Inc.<br>2011-2018 The Bootstrap Authors',
+    '2011-2019 Twitter, Inc.<br>2011-2019 The Bootstrap Authors',
     'CC BY',
     'https://creativecommons.org/licenses/by/3.0/'
   ], [
     'https://github.com/clojure/clojure/blob/master/epl-v10.html'
   ], [
     'CMake',
-    '2000-2018 Kitware, Inc. and Contributors',
+    '2000-2019 Kitware, Inc. and Contributors',
     'BSD',
     'https://cmake.org/licensing/'
   ], [
     '2009-2018 Jeremy Ashkenas',
     'MIT',
     'https://raw.githubusercontent.com/jashkenas/coffeescript/master/LICENSE'
+  ], [
+    'Composer',
+    'Nils Adermann, Jordi Boggiano',
+    'MIT',
+    'https://raw.githubusercontent.com/composer/composer/master/LICENSE'
   ], [
     'Cordova',
     '2012-2018 The Apache Software Foundation',
     'Apache',
     'https://raw.githubusercontent.com/crystal-lang/crystal/master/LICENSE'
   ], [
+    'Cypress',
+    '2017 Cypress.io',
+    'MIT',
+    'https://raw.githubusercontent.com/cypress-io/cypress-documentation/develop/LICENSE.md'
+  ],
+  [
     'D',
     '1999-2018 The D Language Foundation',
     'Boost',
     'https://raw.githubusercontent.com/django/django/master/LICENSE'
   ], [
     'Docker',
-    '2017 Docker, Inc.<br>Docker and the Docker logo are trademarks of Docker, Inc.',
+    '2019 Docker, Inc.<br>Docker and the Docker logo are trademarks of Docker, Inc.',
     'Apache',
     'https://raw.githubusercontent.com/docker/docker.github.io/master/LICENSE'
   ], [
     'https://raw.githubusercontent.com/koajs/koa/master/LICENSE'
   ], [
     'Kotlin',
-    '2010-2018 JetBrains s.r.o.',
+    '2010-2019 JetBrains s.r.o.',
     'Apache',
     'https://raw.githubusercontent.com/JetBrains/kotlin/master/license/LICENSE.txt'
   ], [
     'https://raw.githubusercontent.com/laravel/framework/master/LICENSE.txt'
   ], [
     'Leaflet',
-    '2010-2018 Vladimir Agafonkin<br>&copy; 2010-2011, CloudMade<br>Maps &copy; OpenStreetMap contributors.',
+    '2010-2019 Vladimir Agafonkin<br>&copy; 2010-2011, CloudMade<br>Maps &copy; OpenStreetMap contributors.',
     'BSD',
     'https://raw.githubusercontent.com/Leaflet/Leaflet/master/LICENSE'
   ], [
     'https://github.com/LearnBoost/mongoose/blob/master/README.md#license'
   ], [
     'nginx',
-    '2002-2018 Igor Sysoev<br>&copy; 2011-2018 Nginx, Inc.',
+    '2002-2019 Igor Sysoev<br>&copy; 2011-2019 Nginx, Inc.',
     'BSD',
     'http://nginx.org/LICENSE'
   ], [
     'https://raw.githubusercontent.com/OpenTSDB/opentsdb.net/gh-pages/COPYING.LESSER'
   ], [
     'Padrino',
-    '2010-2016 Padrino',
+    '2010-2019 Padrino',
     'MIT',
     'https://raw.githubusercontent.com/padrino/padrino-framework/master/padrino/LICENSE.txt'
   ], [
     'https://raw.githubusercontent.com/GoogleChrome/puppeteer/master/LICENSE'
   ], [
     'Pygame',
-    'Pygame Developpers',
+    'Pygame Developers',
     'LGPLv2.1',
     'https://raw.githubusercontent.com/pygame/pygame/master/LICENSE'
   ], [
     'MIT',
     'https://raw.githubusercontent.com/rust-lang/rust/master/LICENSE-MIT'
   ], [
+    'Salt Stack',
+    '2019 SaltStack',
+    'Apache',
+    'https://raw.githubusercontent.com/saltstack/salt/develop/LICENSE'
+  ],
+  [
     'Sass',
     '2006-2016 Hampton Catlin, Nathan Weizenbaum, and Chris Eppstein',
     'MIT',
     'https://raw.githubusercontent.com/sass/sass/stable/MIT-LICENSE'
   ], [
+    'Scala',
+    '2002-2019 EPFL, with contributions from Lightbend',
+    'Apache',
+    'https://raw.githubusercontent.com/scala/scala-lang/master/license.md'
+  ],
+  [
     'scikit-image',
     '2011 the scikit-image team',
     'BSD',
     '2013-2018 Evan You, Vue.js contributors',
     'MIT',
     'https://raw.githubusercontent.com/vuejs/vue/master/LICENSE'
+  ], [
+    'Vue Router',
+    '2013-present Evan You',
+    'MIT',
+    'https://raw.githubusercontent.com/vuejs/vue-router/dev/LICENSE'
   ], [
     'Vulkan',
     '2014-2017 Khronos Group Inc.<br>Vulkan and the Vulkan logo are registered trademarks of the Khronos Group Inc.',
     'CC BY',
     'https://creativecommons.org/licenses/by/4.0/'
   ], [
+    'Wordpress',
+    '2003-2019 WordPress Foundation',
+    'GPLv2+',
+    'https://wordpress.org/about/license/'
+  ],
+  [
     'Yarn',
     '2016-present Yarn Contributors',
     'BSD',

assets/javascripts/templates/pages/offline_tmpl.coffee+9−4

   <h2 class="_block-heading">Questions & Answers</h2>
   <dl>
     <dt>How does this work?
-    <dd>Each page is cached as a key-value pair in <a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API">IndexedDB</a> (downloaded from a single file).<br>
+    <dd>Each page is cached as a key-value pair in <a href="https://devdocs.io/dom/indexeddb_api">IndexedDB</a> (downloaded from a single file).<br>
-        The app also uses <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Using_the_application_cache">AppCache</a> and <a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API">localStorage</a> to cache the assets and index files.
+        The app also uses <a href="https://devdocs.io/dom/service_worker_api/using_service_workers">Service Workers</a> and <a href="https://devdocs.io/dom/web_storage_api">localStorage</a> to cache the assets and index files.
     <dt>Can I close the tab/browser?
     <dd>#{canICloseTheTab()}
     <dt>What if I don't update a documentation?
 """
 canICloseTheTab = ->
-  if app.AppCache.isEnabled()
+  if app.ServiceWorker.isEnabled()
     """ Yes! Even offline, you can open a new tab, go to <a href="//devdocs.io">devdocs.io</a>, and everything will work as if you were online (provided you installed all the documentations you want to use beforehand). """
   else
-    """ No. AppCache isn't available in your browser (or is disabled), so loading <a href="//devdocs.io">devdocs.io</a> offline won't work.<br>
+    reason = "aren't available in your browser (or are disabled)"
+    if app.config.env != 'production'
+      reason = "are disabled in your development instance of DevDocs (enable them by setting the <code>ENABLE_SERVICE_WORKER</code> environment variable to <code>true</code>)"
+    """ No. Service Workers #{reason}, so loading <a href="//devdocs.io">devdocs.io</a> offline won't work.<br>
         The current tab will continue to function even when you go offline (provided you installed all the documentations beforehand). """
 app.templates.offlineDoc = (doc, status) ->

assets/javascripts/templates/pages/root_tmpl.coffee.erb+3−3

assets/javascripts/templates/pages/settings_tmpl.coffee+8−0

@@ -15,6 +15,14 @@ app.templates.settingsPage = (settings) -> """ · 14 unmodified lines
15	<input type="checkbox" form="settings" name="layout" value="_sidebar-hidden"#{if settings['_sidebar-hidden'] then ' checked' else ''}>Automatically hide and show the sidebar	15	<input type="checkbox" form="settings" name="layout" value="_sidebar-hidden"#{if settings['_sidebar-hidden'] then ' checked' else ''}>Automatically hide and show the sidebar
16	<small>Tip: drag the edge of the sidebar to resize it.</small>	16	<small>Tip: drag the edge of the sidebar to resize it.</small>
17	</label>	17	</label>
		18	<label class="_settings-label">
		19	<input type="checkbox" form="settings" name="autoInstall" value="_auto-install"#{if settings.autoInstall then ' checked' else ''}>Automatically download documentation for offline use
		20	<small>Only enable this when bandwidth isn't a concern to you.</small>
		21	</label>
		22	<label class="_settings-label">
		23	<input type="checkbox" form="settings" name="analyticsConsent"#{if settings.analyticsConsent then ' checked' else ''}>Enable tracking cookies
		24	<small>With this checked, we enable Google Analytics and Gauges to collect anonymous traffic information.</small>
		25	</label>
18	</div>	26	</div>
19	</div>	27	</div>
20		28

assets/javascripts/tracking.js+27−23

@@ -1,28 +1,32 @@
1	try {	1	try {
2	if (app.config.env == 'production') {	2	if (app.config.env === 'production') {
3	(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]\|\|function(){	3	if (Cookies.get('analyticsConsent') === '1') {
4	(i[r].q=i[r].q\|\|[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),	4	(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]\|\|function(){
5	m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)	5	(i[r].q=i[r].q\|\|[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
6	})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');	6	m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
7	ga('create', 'UA-5544833-12', 'devdocs.io');	7	})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
8	page.track(function() {	8	ga('create', 'UA-5544833-12', 'devdocs.io');
9	ga('send', 'pageview', {	9	page.track(function() {
10	page: location.pathname + location.search + location.hash,	10	ga('send', 'pageview', {
11	dimension1: app.router.context && app.router.context.doc && app.router.context.doc.slug_without_version	11	page: location.pathname + location.search + location.hash,
		12	dimension1: app.router.context && app.router.context.doc && app.router.context.doc.slug_without_version
		13	});
12	});	14	});
13	});
14		15
15	page.track(function() {	16	page.track(function() {
16	if (window._gauges)	17	if (window._gauges)
17	_gauges.push(['track']);	18	_gauges.push(['track']);
18	else	19	else
19	(function() {	20	(function() {
20	var _gauges=_gauges\|\|[];!function(){var a=document.createElement("script");	21	var _gauges=_gauges\|\|[];!function(){var a=document.createElement("script");
21	a.type="text/javascript",a.async=!0,a.id="gauges-tracker",	22	a.type="text/javascript",a.async=!0,a.id="gauges-tracker",
22	a.setAttribute("data-site-id","51c15f82613f5d7819000067"),	23	a.setAttribute("data-site-id","51c15f82613f5d7819000067"),
23	a.src="https://secure.gaug.es/track.js";var b=document.getElementsByTagName("script")[0];	24	a.src="https://secure.gaug.es/track.js";var b=document.getElementsByTagName("script")[0];
24	b.parentNode.insertBefore(a,b)}();	25	b.parentNode.insertBefore(a,b)}();
25	})();	26	})();
26	});	27	});
		28	} else {
		29	resetAnalytics();
		30	}
27	}	31	}
28	} catch(e) { }	32	} catch(e) { }

assets/javascripts/vendor/fastclick.js+2−2

@@ -413,7 +413,7 @@ · 412 unmodified lines
413	// when the user next taps anywhere else on the page, new touchstart and touchend events are dispatched	413	// when the user next taps anywhere else on the page, new touchstart and touchend events are dispatched
414	// with the same identifier as the touch event that previously triggered the click that triggered the alert.	414	// with the same identifier as the touch event that previously triggered the click that triggered the alert.
415	// Sadly, there is an issue on iOS 4 that causes some normal touch events to have the same identifier as an	415	// Sadly, there is an issue on iOS 4 that causes some normal touch events to have the same identifier as an
416	// immediately preceeding touch event (issue #52), so this fix is unavailable on that platform.	416	// immediately preceding touch event (issue #52), so this fix is unavailable on that platform.
417	// Issue 120: touch.identifier is 0 when Chrome dev tools 'Emulate touch events' is set with an iOS device UA string,	417	// Issue 120: touch.identifier is 0 when Chrome dev tools 'Emulate touch events' is set with an iOS device UA string,
418	// which causes all touch events to be ignored. As this block only applies to iOS, and iOS identifiers are always long,	418	// which causes all touch events to be ignored. As this block only applies to iOS, and iOS identifiers are always long,
419	// random integers, it's safe to to continue if the identifier is 0 here.	419	// random integers, it's safe to to continue if the identifier is 0 here.
@@ -805,7 +805,7 @@ · 385 unmodified lines
805	}	805	}
806	}	806	}
807		807
808	// IE11: prefixed -ms-touch-action is no longer supported and it's recomended to use non-prefixed version	808	// IE11: prefixed -ms-touch-action is no longer supported and it's recommended to use non-prefixed version
809	// http://msdn.microsoft.com/en-us/library/windows/apps/Hh767313.aspx	809	// http://msdn.microsoft.com/en-us/library/windows/apps/Hh767313.aspx
810	if (layer.style.touchAction === 'none' \|\| layer.style.touchAction === 'manipulation') {	810	if (layer.style.touchAction === 'none' \|\| layer.style.touchAction === 'manipulation') {
811	return true;	811	return true;

assets/javascripts/vendor/raven.js+1−1

@@ -856,7 +856,7 @@ Raven.prototype = { · 855 unmodified lines
856	},	856	},
857		857
858	_triggerEvent: function(eventType, options) {	858	_triggerEvent: function(eventType, options) {
859	// NOTE: `event` is a native browser thing, so let's avoid conflicting wiht it	859	// NOTE: `event` is a native browser thing, so let's avoid conflicting with it
860	var evt, key;	860	var evt, key;
861		861
862	if (!this._hasDocument) return;	862	if (!this._hasDocument) return;

assets/javascripts/views/content/content.coffee+3−0

@@ -153,6 +153,9 @@ class app.views.Content extends app.View · 152 unmodified lines
153	return	153	return
154		154
155	afterRoute: (route, context) =>	155	afterRoute: (route, context) =>
		156	if route != 'entry' and route != 'type'
		157	resetFavicon()
		158
156	switch route	159	switch route
157	when 'root'	160	when 'root'
158	@show @rootPage	161	@show @rootPage

assets/javascripts/views/content/entry_page.coffee+2−1

     if app.disabledDocs.findBy 'slug', @entry.doc.slug
       @hiddenView = new app.views.HiddenPage @el, @entry
+    setFaviconForDoc(@entry.doc)
     @delay @polyfillMathML
     @trigger 'loaded'
     return
     @render @tmpl('pageLoadError')
     @resetClass()
     @addClass @constructor.errorClass
-    app.appCache?.update()
+    app.serviceWorker?.update()
     return
   cache: ->

assets/javascripts/views/content/settings_page.coffee+10−11

 class app.views.SettingsPage extends app.View
-  LAYOUTS = ['_max-width', '_sidebar-hidden', '_native-scrollbars']
-  SIDEBAR_HIDDEN_LAYOUT = '_sidebar-hidden'
   @className: '_static'
   @events:
     settings.dark = app.settings.get('dark')
     settings.smoothScroll = !app.settings.get('fastScroll')
     settings.arrowScroll = app.settings.get('arrowScroll')
-    settings[layout] = app.settings.hasLayout(layout) for layout in LAYOUTS
+    settings.autoInstall = app.settings.get('autoInstall')
+    settings.analyticsConsent = app.settings.get('analyticsConsent')
+    settings[layout] = app.settings.hasLayout(layout) for layout in app.settings.LAYOUTS
     settings
   getTitle: ->
     'Preferences'
   toggleDark: (enable) ->
-    html = document.documentElement
-    html.classList.toggle('_theme-default')
-    html.classList.toggle('_theme-dark')
     app.settings.set('dark', !!enable)
-    app.appCache?.updateInBackground()
     return
   toggleLayout: (layout, enable) ->
-    document.body.classList[if enable then 'add' else 'remove'](layout) unless layout is SIDEBAR_HIDDEN_LAYOUT
-    document.body.classList[if $.overlayScrollbarsEnabled() then 'add' else 'remove']('_overlay-scrollbars')
     app.settings.setLayout(layout, enable)
-    app.appCache?.updateInBackground()
     return
   toggleSmoothScroll: (enable) ->
     app.settings.set('fastScroll', !enable)
     return
+  toggleAnalyticsConsent: (enable) ->
+    app.settings.set('analyticsConsent', if enable then '1' else '0')
+    resetAnalytics() unless enable
+    return
   toggle: (name, enable) ->
     app.settings.set(name, enable)
     return
         @toggleSmoothScroll input.checked
       when 'import'
         @import input.files[0], input
+      when 'analyticsConsent'
+        @toggleAnalyticsConsent input.checked
       else
         @toggle input.name, input.checked
     return

assets/javascripts/views/content/type_page.coffee+1−0

@@ -9,6 +9,7 @@ class app.views.TypePage extends app.View · 8 unmodified lines
9		9
10	render: (@type) ->	10	render: (@type) ->
11	@html @tmpl('typePage', @type)	11	@html @tmpl('typePage', @type)
		12	setFaviconForDoc(@type.doc)
12	return	13	return
13		14
14	getTitle: ->	15	getTitle: ->

assets/javascripts/views/layout/document.coffee+7−5

@@ -75,9 +75,11 @@ class app.views.Document extends app.View · 74 unmodified lines
75	return unless target.hasAttribute('data-behavior')	75	return unless target.hasAttribute('data-behavior')
76	$.stopEvent(event)	76	$.stopEvent(event)
77	switch target.getAttribute('data-behavior')	77	switch target.getAttribute('data-behavior')
78	when 'back' then history.back()	78	when 'back' then history.back()
79	when 'reload' then window.location.reload()	79	when 'reload' then window.location.reload()
80	when 'reboot' then app.reboot()	80	when 'reboot' then app.reboot()
81	when 'hard-reload' then app.reload()	81	when 'hard-reload' then app.reload()
82	when 'reset' then app.reset() if confirm('Are you sure you want to reset DevDocs?')	82	when 'reset' then app.reset() if confirm('Are you sure you want to reset DevDocs?')
		83	when 'accept-analytics' then Cookies.set('analyticsConsent', '1', expires: 1e8) && app.reboot()
		84	when 'decline-analytics' then Cookies.set('analyticsConsent', '0', expires: 1e8) && app.reboot()
83	return	85	return

assets/javascripts/views/layout/resizer.coffee+2−9

   init: ->
     @el.setAttribute('draggable', 'true')
     @appendTo $('._app')
-    @style = $('style[data-resizer]')
-    @size = @style.getAttribute('data-size')
     return
   MIN = 260
     return unless value > 0
     value = Math.min(Math.max(Math.round(value), MIN), MAX)
     newSize = "#{value}px"
-    @style.innerHTML = @style.innerHTML.replace(new RegExp(@size, 'g'), newSize)
+    document.documentElement.style.setProperty('--sidebarWidth', newSize)
-    @size = newSize
+    app.settings.setSize(value) if save
-    if save
-      app.settings.setSize(value)
-      app.appCache?.updateInBackground()
     return
   onDragStart: (event) =>
-    @style.removeAttribute('disabled')
     event.dataTransfer.effectAllowed = 'link'
     event.dataTransfer.setData('Text', '')
     $.on(window, 'dragover', @onDrag)

assets/javascripts/views/layout/settings.coffee+1−9

     if super
       @render()
       document.body.classList.remove(SIDEBAR_HIDDEN_LAYOUT)
-      app.appCache?.on 'progress', @onAppCacheProgress
     return
   deactivate: ->
       @resetClass()
       @docPicker.detach()
       document.body.classList.add(SIDEBAR_HIDDEN_LAYOUT) if app.settings.hasLayout(SIDEBAR_HIDDEN_LAYOUT)
-      app.appCache?.off 'progress', @onAppCacheProgress
     return
   render: ->
         docs = @docPicker.getSelectedDocs()
         app.settings.setDocs(docs)
-      @saveBtn.textContent = if app.appCache then 'Downloading\u2026' else 'Saving\u2026'
+      @saveBtn.textContent = 'Saving\u2026'
       disabledDocs = new app.collections.Docs(doc for doc in app.docs.all() when docs.indexOf(doc.slug) is -1)
       disabledDocs.uninstall ->
         app.db.migrate()
       $.stopEvent(event)
       app.router.show '/'
     return
-  onAppCacheProgress: (event) =>
-    if event.lengthComputable
-      percentage = Math.round event.loaded * 100 / event.total
-      @saveBtn.textContent = "Downloading\u2026 (#{percentage}%)"
-    return

assets/javascripts/views/search/search.coffee+8−0

       .on 'results', @onResults
       .on 'end', @onEnd
+    @scope
+      .on 'change', @onScopeChange
     app.on 'ready', @onReady
     $.on window, 'hashchange', @searchUrl
     $.on window, 'focus', @onWindowFocus
     $.stopEvent(event)
     return
+  onScopeChange: =>
+    @value = ''
+    @onInput()
+    return
   afterRoute: (name, context) =>
     return if app.shortcuts.eventInProgress?.name is 'escape'
     @reset(true) if not context.init and app.router.isIndex()

assets/javascripts/views/search/search_scope.coffee+23−5

     tag:   '._search-tag'
   @events:
+    click: 'onClick'
     keydown: 'onKeydown'
+    textInput: 'onTextInput'
   @routes:
     after: 'afterRoute'
     @trigger 'change', null, previousDoc
     return
+  doScopeSearch: (event) =>
+    @search @input.value[0...@input.selectionStart]
+    $.stopEvent(event) if @doc
+    return
+  onClick: (event) =>
+    if event.target is @tag
+      @reset()
+      $.stopEvent(event)
+    return
   onKeydown: (event) =>
     if event.which is 8 # backspace
-      if @doc and not @input.value
+      if @doc and @input.selectionEnd is 0
-        $.stopEvent(event)
         @reset()
-    else if not @doc and @input.value
+        $.stopEvent(event)
+    else if not @doc and @input.value and not $.isChromeForAndroid()
       return if event.ctrlKey or event.metaKey or event.altKey or event.shiftKey
       if event.which is 9 or # tab
          (event.which is 32 and app.isMobile()) # space
-        @search @input.value[0...@input.selectionStart]
+        @doScopeSearch(event)
-        $.stopEvent(event) if @doc
+    return
+  onTextInput: (event) =>
+    return unless $.isChromeForAndroid()
+    if not @doc and @input.value and event.data == ' '
+      @doScopeSearch(event)
     return
   extractHashValue: ->

assets/javascripts/views/sidebar/sidebar.coffee+1−1

@@ -28,7 +28,7 @@ class app.views.Sidebar extends app.View · 27 unmodified lines
28		28
29	app.on 'ready', @onReady	29	app.on 'ready', @onReady
30		30
31	$.on document.documentElement, 'mouseleave', (event) => @display() if event.clientX < 10	31	$.on document.documentElement, 'mouseleave', (event) => @display() unless event.clientX <= 0
32	$.on document.documentElement, 'mouseenter', => @resetDisplay(forceNoHover: false)	32	$.on document.documentElement, 'mouseenter', => @resetDisplay(forceNoHover: false)
33	return	33	return
34		34

assets/stylesheets/application.css.scss+7−5

-//= depend_on docs-1.png
+//= depend_on sprites/docs.png
-//= depend_on docs-1@2x.png
+//= depend_on sprites/docs@2x.png
-//= depend_on docs-2.png
+//= depend_on sprites/docs.json
-//= depend_on docs-2@2x.png
 /*!
- * Copyright 2013-2018 Thibaut Courouble and other contributors
+ * Copyright 2013-2019 Thibaut Courouble and other contributors
  *
  * This source code is licensed under the terms of the Mozilla
  * Public License, v. 2.0, a copy of which may be obtained at:
         'pages/coffeescript',
         'pages/cordova',
         'pages/crystal',
+        'pages/cypress',
         'pages/d',
         'pages/d3',
         'pages/dart',
         'pages/rfc',
         'pages/rubydoc',
         'pages/rust',
+        'pages/scala',
         'pages/sinon',
         'pages/socketio',
         'pages/sphinx',
         'pages/underscore',
         'pages/vue',
         'pages/webpack',
+        'pages/wordpress',
         'pages/yard',
         'pages/yii';

assets/stylesheets/components/_fail.scss+0−2

@@ -32,5 +32,3 @@ · 31 unmodified lines
32	}	32	}
33		33
34	._fail-text:last-child { margin: 0; }	34	._fail-text:last-child { margin: 0; }
35
36	._fail-link { float: right; }

assets/stylesheets/components/_header.scss+1−0

@@ -215,5 +215,6 @@ · 214 unmodified lines
215	color: var(--textColorLight);	215	color: var(--textColorLight);
216	background: var(--searchTagBackground);	216	background: var(--searchTagBackground);
217	border-radius: 2px;	217	border-radius: 2px;
		218	cursor: pointer;
218	@extend %truncate-text;	219	@extend %truncate-text;
219	}	220	}

assets/stylesheets/components/_notif.scss+4−0

@@ -134,3 +134,7 @@ · 133 unmodified lines
134		134
135	._notif-info { color: var(--textColorLight); }	135	._notif-info { color: var(--textColorLight); }
136	}	136	}
		137
		138	._notif-right {
		139	float: right;
		140	}

assets/stylesheets/global/_icons.scss+0−182

@@ -1,182 +0,0 @@
1	%svg-icon {
2	display: inline-block;
3	vertical-align: top;
4	width: 1rem;
5	height: 1rem;
6	pointer-events: none;
7	fill: currentColor;
8	}
9
10	%doc-icon {
11	content: '';
12	display: block;
13	width: 1rem;
14	height: 1rem;
15	background-image: image-url('docs-1.png');
16	background-size: 10rem 10rem;
17	}
18
19	%doc-icon-2 { background-image: image-url('docs-2.png') !important; }
20
21	@media (-webkit-min-device-pixel-ratio: 1.5), (min-resolution: 144dpi) {
22	%doc-icon { background-image: image-url('docs-1@2x.png'); }
23	%doc-icon-2 { background-image: image-url('docs-2@2x.png') !important; }
24	}
25
26	html._theme-dark {
27	%darkIconFix {
28	filter: invert(100%) grayscale(100%);
29	-webkit-filter: invert(100%) grayscale(100%);
30	}
31	}
32
33	._icon-jest:before { background-position: 0 0; }
34	._icon-liquid:before { background-position: -1rem 0; }
35	._icon-openjdk:before { background-position: -2rem 0; }
36	._icon-codeceptjs:before { background-position: -3rem 0; }
37	._icon-codeception:before { background-position: -4rem 0; }
38	._icon-sqlite:before { background-position: -5rem 0; @extend %darkIconFix !optional; }
39	._icon-async:before { background-position: -6rem 0; @extend %darkIconFix !optional; }
40	._icon-http:before { background-position: -7rem 0; @extend %darkIconFix !optional; }
41	._icon-jquery:before { background-position: -8rem 0; @extend %darkIconFix !optional; }
42	._icon-underscore:before { background-position: -9rem 0; @extend %darkIconFix !optional; }
43	._icon-html:before { background-position: 0 -1rem; }
44	._icon-css:before { background-position: -1rem -1rem; }
45	._icon-dom:before { background-position: -2rem -1rem; }
46	._icon-dom_events:before { background-position: -3rem -1rem; }
47	._icon-javascript:before { background-position: -4rem -1rem; }
48	._icon-backbone:before { background-position: -5rem -1rem; @extend %darkIconFix !optional; }
49	._icon-node:before,
50	._icon-node_lts:before { background-position: -6rem -1rem; }
51	._icon-sass:before { background-position: -7rem -1rem; }
52	._icon-less:before { background-position: -8rem -1rem; }
53	._icon-angularjs:before { background-position: -9rem -1rem; }
54	._icon-coffeescript:before { background-position: 0 -2rem; @extend %darkIconFix !optional; }
55	._icon-ember:before { background-position: -1rem -2rem; }
56	._icon-yarn:before { background-position: -2rem -2rem; }
57	._icon-immutable:before { background-position: -3rem -2rem; @extend %darkIconFix !optional; }
58	._icon-jqueryui:before { background-position: -4rem -2rem; }
59	._icon-jquerymobile:before { background-position: -5rem -2rem; }
60	._icon-lodash:before { background-position: -6rem -2rem; }
61	._icon-php:before { background-position: -7rem -2rem; }
62	._icon-ruby:before,
63	._icon-minitest:before { background-position: -8rem -2rem; }
64	._icon-rails:before { background-position: -9rem -2rem; }
65	._icon-python:before,
66	._icon-python2:before { background-position: 0 -3rem; }
67	._icon-git:before { background-position: -1rem -3rem; }
68	._icon-redis:before { background-position: -2rem -3rem; }
69	._icon-postgresql:before { background-position: -3rem -3rem; }
70	._icon-d3:before { background-position: -4rem -3rem; }
71	._icon-knockout:before { background-position: -5rem -3rem; }
72	._icon-moment:before { background-position: -6rem -3rem; @extend %darkIconFix !optional; }
73	._icon-c:before { background-position: -7rem -3rem; }
74	._icon-statsmodels:before { background-position: -8rem -3rem; }
75	._icon-yii:before,
76	._icon-yii1:before { background-position: -9rem -3rem; }
77	._icon-cpp:before { background-position: 0 -4rem; }
78	._icon-go:before { background-position: -1rem -4rem; }
79	._icon-express:before { background-position: -2rem -4rem; }
80	._icon-grunt:before { background-position: -3rem -4rem; }
81	._icon-rust:before { background-position: -4rem -4rem; @extend %darkIconFix !optional; }
82	._icon-laravel:before { background-position: -5rem -4rem; }
83	._icon-haskell:before { background-position: -6rem -4rem; }
84	._icon-requirejs:before { background-position: -7rem -4rem; }
85	._icon-chai:before { background-position: -8rem -4rem; }
86	._icon-sinon:before { background-position: -9rem -4rem; }
87	._icon-cordova:before { background-position: 0 -5rem; }
88	._icon-markdown:before { background-position: -1rem -5rem; @extend %darkIconFix !optional; }
89	._icon-django:before { background-position: -2rem -5rem; }
90	._icon-xslt_xpath:before { background-position: -3rem -5rem; }
91	._icon-nginx:before,
92	._icon-nginx_lua_module:before { background-position: -4rem -5rem; }
93	._icon-svg:before { background-position: -5rem -5rem; }
94	._icon-marionette:before { background-position: -6rem -5rem; }
95	._icon-jsdoc:before,
96	._icon-koa:before,
97	._icon-graphite:before,
98	._icon-mongoose:before { background-position: -7rem -5rem; }
99	._icon-phpunit:before { background-position: -8rem -5rem; }
100	._icon-nokogiri:before { background-position: -9rem -5rem; @extend %darkIconFix !optional; }
101	._icon-rethinkdb:before { background-position: 0 -6rem; }
102	._icon-react:before { background-position: -1rem -6rem; }
103	._icon-socketio:before { background-position: -2rem -6rem; }
104	._icon-modernizr:before { background-position: -3rem -6rem; }
105	._icon-bower:before { background-position: -4rem -6rem; }
106	._icon-fish:before { background-position: -5rem -6rem; @extend %darkIconFix !optional; }
107	._icon-scikit_image:before { background-position: -6rem -6rem; }
108	._icon-twig:before { background-position: -7rem -6rem; }
109	._icon-pandas:before { background-position: -8rem -6rem; }
110	._icon-scikit_learn:before { background-position: -9rem -6rem; }
111	._icon-bottle:before { background-position: 0 -7rem; }
112	._icon-docker:before { background-position: -1rem -7rem; }
113	._icon-cakephp:before { background-position: -2rem -7rem; }
114	._icon-lua:before { background-position: -3rem -7rem; @extend %darkIconFix !optional; }
115	._icon-clojure:before { background-position: -4rem -7rem; }
116	._icon-symfony:before { background-position: -5rem -7rem; }
117	._icon-mocha:before { background-position: -6rem -7rem; }
118	._icon-meteor:before { background-position: -7rem -7rem; @extend %darkIconFix !optional; }
119	._icon-npm:before { background-position: -8rem -7rem; }
120	._icon-apache_http_server:before { background-position: -9rem -7rem; }
121	._icon-drupal:before { background-position: 0 -8rem; }
122	._icon-webpack:before { background-position: -1rem -8rem; }
123	._icon-phaser:before { background-position: -2rem -8rem; }
124	._icon-vue:before { background-position: -3rem -8rem; }
125	._icon-opentsdb:before { background-position: -4rem -8rem; }
126	._icon-q:before { background-position: -5rem -8rem; }
127	._icon-crystal:before { background-position: -6rem -8rem; @extend %darkIconFix !optional; }
128	._icon-julia:before { background-position: -7rem -8rem; @extend %darkIconFix !optional; }
129	._icon-redux:before { background-position: -8rem -8rem; @extend %darkIconFix !optional; }
130	._icon-bootstrap:before { background-position: -9rem -8rem; }
131	._icon-react_native:before { background-position: 0 -9rem; }
132	._icon-phalcon:before { background-position: -1rem -9rem; }
133	._icon-matplotlib:before { background-position: -2rem -9rem; }
134	._icon-cmake:before { background-position: -3rem -9rem; }
135	._icon-elixir:before { background-position: -4rem -9rem; @extend %darkIconFix !optional; }
136	._icon-vagrant:before { background-position: -5rem -9rem; }
137	._icon-dojo:before { background-position: -6rem -9rem; }
138	._icon-flow:before { background-position: -7rem -9rem; }
139	._icon-relay:before { background-position: -8rem -9rem; }
140	._icon-phoenix:before { background-position: -9rem -9rem; }
141
142	._icon-tcl_tk:before { background-position: 0 0; @extend %doc-icon-2; }
143	._icon-erlang:before { background-position: -1rem 0; @extend %doc-icon-2; }
144	._icon-chef:before { background-position: -2rem 0; @extend %doc-icon-2; }
145	._icon-ramda:before { background-position: -3rem 0; @extend %doc-icon-2; @extend %darkIconFix !optional; }
146	._icon-codeigniter:before { background-position: -4rem 0; @extend %doc-icon-2; @extend %darkIconFix !optional; }
147	._icon-influxdata:before { background-position: -5rem 0; @extend %doc-icon-2; @extend %darkIconFix !optional; }
148	._icon-tensorflow:before { background-position: -6rem 0; @extend %doc-icon-2; }
149	._icon-haxe:before { background-position: -7rem 0; @extend %doc-icon-2; }
150	._icon-ansible:before { background-position: -8rem 0; @extend %doc-icon-2; @extend %darkIconFix !optional; }
151	._icon-typescript:before { background-position: -9rem 0; @extend %doc-icon-2; }
152	._icon-browser_support_tables:before { background-position: 0rem -1rem; @extend %doc-icon-2; }
153	._icon-gnu_fortran:before { background-position: -1rem -1rem; @extend %doc-icon-2; }
154	._icon-gcc:before { background-position: -2rem -1rem; @extend %doc-icon-2; }
155	._icon-perl:before { background-position: -3rem -1rem; @extend %doc-icon-2; }
156	._icon-apache_pig:before { background-position: -4rem -1rem; @extend %doc-icon-2; }
157	._icon-numpy:before { background-position: -5rem -1rem; @extend %doc-icon-2; }
158	._icon-kotlin:before { background-position: -6rem -1rem; @extend %doc-icon-2; }
159	._icon-padrino:before { background-position: -7rem -1rem; @extend %doc-icon-2; }
160	._icon-angular:before { background-position: -8rem -1rem; @extend %doc-icon-2; }
161	._icon-love:before { background-position: -9rem -1rem; @extend %doc-icon-2; }
162	._icon-jasmine:before { background-position: 0 -2rem; @extend %doc-icon-2; }
163	._icon-pug:before { background-position: -1rem -2rem; @extend %doc-icon-2; }
164	._icon-electron:before { background-position: -2rem -2rem; @extend %doc-icon-2; }
165	._icon-falcon:before { background-position: -3rem -2rem; @extend %doc-icon-2; }
166	._icon-godot:before { background-position: -4rem -2rem; @extend %doc-icon-2; }
167	._icon-nim:before { background-position: -5rem -2rem; @extend %doc-icon-2; @extend %darkIconFix !optional; }
168	._icon-vulkan:before { background-position: -6rem -2rem; @extend %doc-icon-2; @extend %darkIconFix !optional; }
169	._icon-d:before { background-position: -7rem -2rem; @extend %doc-icon-2; }
170	._icon-bluebird:before { background-position: -8rem -2rem; @extend %doc-icon-2; }
171	._icon-eslint:before { background-position: -9rem -2rem; @extend %doc-icon-2; }
172	._icon-homebrew:before { background-position: 0 -3rem; @extend %doc-icon-2; }
173	._icon-jekyll:before { background-position: -1rem -3rem; @extend %doc-icon-2; }
174	._icon-babel:before { background-position: -2rem -3rem; @extend %doc-icon-2; }
175	._icon-leaflet:before { background-position: -3rem -3rem; @extend %doc-icon-2; }
176	._icon-terraform:before { background-position: -4rem -3rem; @extend %doc-icon-2; }
177	._icon-pygame:before { background-position: -5rem -3rem; @extend %doc-icon-2; }
178	._icon-bash:before { background-position: -6rem -3rem; @extend %doc-icon-2; }
179	._icon-dart:before { background-position: -7rem -3rem; @extend %doc-icon-2; }
180	._icon-qt:before { background-position: -8rem -3rem; @extend %doc-icon-2; }
181	._icon-puppeteer:before { background-position: -9rem -3rem; @extend %doc-icon-2; }
182	._icon-handlebars:before { background-position: 0 -4rem; @extend %doc-icon-2; @extend %darkIconFix !optional; }

assets/stylesheets/global/_icons.scss.erb+43−0

@@ -0,0 +1,43 @@
	1	<% manifest = JSON.parse(File.read('assets/images/sprites/docs.json')) %>
	2
	3	%svg-icon {
	4	display: inline-block;
	5	vertical-align: top;
	6	width: 1rem;
	7	height: 1rem;
	8	pointer-events: none;
	9	fill: currentColor;
	10	}
	11
	12	%doc-icon {
	13	content: '';
	14	display: block;
	15	width: 1rem;
	16	height: 1rem;
	17	background-image: image-url('sprites/docs.png');
	18	background-size: <%= manifest['icons_per_row'] %>rem <%= manifest['icons_per_row'] %>rem;
	19	}
	20
	21	@media (-webkit-min-device-pixel-ratio: 1.5), (min-resolution: 144dpi) {
	22	%doc-icon { background-image: image-url('sprites/docs@2x.png'); }
	23	}
	24
	25	html._theme-dark {
	26	%darkIconFix {
	27	filter: invert(100%) grayscale(100%);
	28	-webkit-filter: invert(100%) grayscale(100%);
	29	}
	30	}
	31
	32	<%=
	33	items = []
	34
	35	manifest['items'].each do \|item\|
	36	rules = []
	37	rules << "background-position: -#{item['col']}rem -#{item['row']}rem;"
	38	rules << "@extend %darkIconFix !optional;" if item['dark_icon_fix']
	39	items << "._icon-#{item['type']}:before { #{rules.join(' ')} }"
	40	end
	41
	42	items.join('')
	43	%>

assets/stylesheets/pages/_cypress.scss+21−0

@@ -0,0 +1,21 @@
	1	._cypress {
	2	@extend %simple;
	3
	4	.note {
	5	h1 {
	6	margin-left: inherit
	7	}
	8
	9	&.danger {
	10	@extend %note-red
	11	}
	12
	13	&.info {
	14	@extend %note-blue
	15	}
	16
	17	&.success {
	18	@extend %note-green
	19	}
	20	}
	21	}

assets/stylesheets/pages/_mdn.scss+25−0

@@ -30,6 +30,7 @@ · 29 unmodified lines
30	.notice,	30	.notice,
31	.warning,	31	.warning,
32	.overheadIndicator,	32	.overheadIndicator,
		33	.blockIndicator,
33	.syntaxbox, // CSS, JavaScript	34	.syntaxbox, // CSS, JavaScript
34	.twopartsyntaxbox, // CSS	35	.twopartsyntaxbox, // CSS
35	.inheritsbox, // JavaScript	36	.inheritsbox, // JavaScript
@@ -104,4 +105,28 @@ · 68 unmodified lines
104	.cleared { clear: both; } // CSS/box-shadow	105	.cleared { clear: both; } // CSS/box-shadow
105		106
106	code > strong { font-weight: normal; }	107	code > strong { font-weight: normal; }
		108
		109	// Compatibility tablees
		110
		111	.bc-github-link {
		112	float: right;
		113	font-size: .75rem;
		114	}
		115
		116	.bc-supports-yes, .bc-supports-yes + dd, .bc-supports-yes + dd + dd { background: var(--noteGreenBackground); }
		117	.bc-supports-partial, .bc-supports-partial + dd, .bc-supports-partial + dd + dd { background: var(--noteOrangeBackground); }
		118	.bc-supports-no, .bc-supports-no + dd, .bc-supports-no + dd + dd { background: var(--noteRedBackground); }
		119
		120	.bc-table {
		121	min-width: 100%;
		122
		123	dl {
		124	margin: .25rem 0 0;
		125	padding: .25rem 0 0;
		126	font-size: .75rem;
		127	border-top: 1px solid var(--boxBorder);
		128	}
		129
		130	dd { margin: 0; }
		131	}
107	}	132	}

assets/stylesheets/pages/_rdoc.scss+0−11

@@ -33,19 +33,8 @@ · 32 unmodified lines
33	}	33	}
34	}	34	}
35		35
36	.method-description { position: relative; }
37
38	.method-source-code {	36	.method-source-code {
39	display: none;	37	display: none;
40	position: absolute;
41	z-index: 1;
42	top: 0;
43	left: -1em;
44	right: 0;
45	background: var(--contentBackground);
46	box-shadow: 0 1em 1em 1em var(--contentBackground);
47
48	> pre { margin: 0; }
49	}	38	}
50		39
51	// Rails guides	40	// Rails guides

assets/stylesheets/pages/_scala.scss+4−0

@@ -0,0 +1,4 @@
	1	._scala {
	2	@extend %simple;
	3	.deprecated { @extend %label-red; }
	4	}

assets/stylesheets/pages/_wordpress.scss+15−0

@@ -0,0 +1,15 @@
	1	._wordpress {
	2	@extend %simple;
	3
	4	.breadcrumbs {
	5	display: none;
	6	}
	7
	8	.callout-warning {
	9	@extend %note, %note-red;
	10	}
	11
	12	.callout-alert {
	13	@extend %note, %note-orange;
	14	}
	15	}

docs/adding-docs.md+24−0

@@ -0,0 +1,24 @@
	1	Adding a documentation may look like a daunting task but once you get the hang of it, it's actually quite simple. Don't hesitate to ask for help [in Gitter](https://gitter.im/FreeCodeCamp/DevDocs) if you ever get stuck.
	2
	3	Note: please read the [contributing guidelines](../.github/CONTRIBUTING.md) before submitting a new documentation.
	4
	5	1. Create a subclass of `Docs::UrlScraper` or `Docs::FileScraper` in the `lib/docs/scrapers/` directory. Its name should be the [PascalCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename (e.g. `my_doc` → `MyDoc`)
	6	2. Add the appropriate class attributes and filter options (see the [Scraper Reference](./scraper-reference.md) page).
	7	3. Check that the scraper is listed in `thor docs:list`.
	8	4. Create filters specific to the scraper in the `lib/docs/filters/[my_doc]/` directory and add them to the class's [filter stacks](./scraper-reference.md#filter-stacks). You may create any number of filters but will need at least the following two:
	9	* A [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter whose task is to clean the HTML markup (e.g. adding `id` attributes to headings) and remove everything superfluous and/or nonessential.
	10	* An [`Entries`](./filter-reference.md#entriesfilter) filter whose task is to determine the pages' metadata (the list of entries, each with a name, type and path).
	11	The [Filter Reference](./filter-reference.md) page has all the details about filters.
	12	5. Using the `thor docs:page [my_doc] [path]` command, check that the scraper works properly. Files will appear in the `public/docs/[my_doc]/` directory (but not inside the app as the command doesn't touch the index). `path` in this case refers to either the remote path (if using `UrlScraper`) or the local path (if using `FileScraper`).
	13	6. Generate the full documentation using the `thor docs:generate [my_doc] --force` command. Additionally, you can use the `--verbose` option to see which files are being created/updated/deleted (useful to see what changed since the last run), and the `--debug` option to see which URLs are being requested and added to the queue (useful to pin down which page adds unwanted URLs to the queue).
	14	7. Start the server, open the app, enable the documentation, and see how everything plays out.
	15	8. Tweak the scraper/filters and repeat 5) and 6) until the pages and metadata are ok.
	16	9. To customize the pages' styling, create an SCSS file in the `assets/stylesheets/pages/` directory and import it in both `application.css.scss` AND `application-dark.css.scss`. Both the file and CSS class should be named `_[type]` where [type] is equal to the scraper's `type` attribute (documentations with the same type share the same custom CSS and JS). Setting the type to `simple` will apply the general styling rules in `assets/stylesheets/pages/_simple.scss`, which can be used for documentations where little to no CSS changes are needed.
	17	10. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works).
	18	11. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. The icon spritesheet is automatically generated when you (re)start your local DevDocs instance.
	19	12. Add the documentation's copyright details to the list in `assets/javascripts/templates/pages/about_tmpl.coffee`. This is the data shown in the table on the [about](https://devdocs.io/about) page, and is ordered alphabetically. Simply copying an existing item, placing it in the right slot and updating the values to match the new scraper will do the job.
	20	13. Ensure `thor updates:check [my_doc]` shows the correct latest version.
	21
	22	If the documentation includes more than a few hundreds pages and is available for download, try to scrape it locally (e.g. using `FileScraper`). It'll make the development process much faster and avoids putting too much load on the source site. (It's not a problem if your scraper is coupled to your local setup, just explain how it works in your pull request.)
	23
	24	Finally, try to document your scraper and filters' behavior as much as possible using comments (e.g. why some URLs are ignored, HTML markup removed, metadata that way, etc.). It'll make updating the documentation much easier.

docs/filter-reference.md+224−0

@@ -0,0 +1,224 @@
	1	Table of contents:
	2
	3	* [Overview](#overview)
	4	* [Instance methods](#instance-methods)
	5	* [Core filters](#core-filters)
	6	* [Custom filters](#custom-filters)
	7	- [CleanHtmlFilter](#cleanhtmlfilter)
	8	- [EntriesFilter](#entriesfilter)
	9
	10	## Overview
	11
	12	Filters use the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. They take an HTML string or [Nokogiri](http://nokogiri.org/) node as input, optionally perform modifications and/or extract information from it, and then outputs the result. Together they form a pipeline where each filter hands its output to the next filter's input. Every documentation page passes through this pipeline before being copied on the local filesystem.
	13
	14	Filters are subclasses of the [`Docs::Filter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter.rb) class and require a `call` method. A basic implementation looks like this:
	15
	16	```ruby
	17	module Docs
	18	class CustomFilter < Filter
	19	def call
	20	doc
	21	end
	22	end
	23	end
	24	```
	25
	26	Filters which manipulate the Nokogiri node object (`doc` and related methods) are _HTML filters_ and must not manipulate the HTML string (`html`). Vice-versa, filters which manipulate the string representation of the document are _text filters_ and must not manipulate the Nokogiri node object. The two types are divided into two stacks within the scrapers. These stacks are then combined into a pipeline that calls the HTML filters before the text filters (more details [here](./scraper-reference.md#filter-stacks)). This is to avoid parsing the document multiple times.
	27
	28	The `call` method must return either `doc` or `html`, depending on the type of filter.
	29
	30	## Instance methods
	31
	32	* `doc` [Nokogiri::XML::Node]
	33	The Nokogiri representation of the container element.
	34	See [Nokogiri's API docs](http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Node) for the list of available methods.
	35
	36	* `html` [String]
	37	The string representation of the container element.
	38
	39	* `context` [Hash] (frozen)
	40	The scraper's `options` along with a few additional keys: `:base_url`, `:root_url`, `:root_page` and `:url`.
	41
	42	* `result` [Hash]
	43	Used to store the page's metadata and pass back information to the scraper.
	44	Possible keys:
	45
	46	- `:path` — the page's normalized path
	47	- `:store_path` — the path where the page will be stored (equal to `:path` with `.html` at the end)
	48	- `:internal_urls` — the list of distinct internal URLs found within the page
	49	- `:entries` — the [`Entry`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/entry.rb) objects to add to the index
	50
	51	* `css`, `at_css`, `xpath`, `at_xpath`
	52	Shortcuts for `doc.css`, `doc.xpath`, etc.
	53
	54	* `base_url`, `current_url`, `root_url` [Docs::URL]
	55	Shortcuts for `context[:base_url]`, `context[:url]`, and `context[:root_url]` respectively.
	56
	57	* `root_path` [String]
	58	Shortcut for `context[:root_path]`.
	59
	60	* `subpath` [String]
	61	The sub-path from the base URL of the current URL.
	62	_Example: if `base_url` equals `example.com/docs` and `current_url` equals `example.com/docs/file?raw`, the returned value is `/file`._
	63
	64	* `slug` [String]
	65	The `subpath` removed of any leading slash or `.html` extension.
	66	_Example: if `subpath` equals `/dir/file.html`, the returned value is `dir/file`._
	67
	68	* `root_page?` [Boolean]
	69	Returns `true` if the current page is the root page.
	70
	71	* `initial_page?` [Boolean]
	72	Returns `true` if the current page is the root page or its subpath is one of the scraper's `initial_paths`.
	73
	74	## Core filters
	75
	76	* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
	77	* [`CleanHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
	78	* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
	79	* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
	80	* [`NormalizePathsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
	81	* [`CleanLocalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
	82	* [`InnerHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
	83	* [`CleanTextFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
	84	* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
	85	* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb) — prepends the document with a title (disabled by default)
	86	* [`EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) — abstract filter for extracting the page's metadata
	87
	88	## Custom filters
	89
	90	Scrapers can have any number of custom filters but require at least the two described below.
	91
	92	Note: filters are located in the [`lib/docs/filters`](https://github.com/Thibaut/devdocs/tree/master/lib/docs/filters/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
	93
	94	### `CleanHtmlFilter`
	95
	96	The `CleanHtml` filter is tasked with cleaning the HTML markup where necessary and removing anything superfluous or nonessential. Only the core documentation should remain at the end.
	97
	98	Nokogiri's many jQuery-like methods make it easy to search and modify elements — see the [API docs](http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Node).
	99
	100	Here's an example implementation that covers the most common use-cases:
	101
	102	```ruby
	103	module Docs
	104	class MyScraper
	105	class CleanHtmlFilter < Filter
	106	def call
	107	css('hr').remove
	108	css('#changelog').remove if root_page?
	109
	110	# Set id attributes on <h3> instead of an empty <a>
	111	css('h3').each do \|node\|
	112	node['id'] = node.at_css('a')['id']
	113	end
	114
	115	# Make proper table headers
	116	css('td.header').each do \|node\|
	117	node.name = 'th'
	118	end
	119
	120	# Remove code highlighting
	121	css('pre').each do \|node\|
	122	node.content = node.content
	123	end
	124
	125	doc
	126	end
	127	end
	128	end
	129	end
	130	```
	131
	132	Notes:
	133
	134	* Empty elements will be automatically removed by the core `CleanTextFilter` later in the pipeline's execution.
	135	* Although the goal is to end up with a clean version of the page, try to keep the number of modifications to a minimum, so as to make the code easier to maintain. Custom CSS is the preferred way of normalizing the pages (except for hiding stuff which should always be done by removing the markup).
	136	* Try to document your filter's behavior as much as possible, particularly modifications that apply only to a subset of pages. It'll make updating the documentation easier.
	137
	138	### `EntriesFilter`
	139
	140	The `Entries` filter is responsible for extracting the page's metadata, represented by a set of _entries_, each with a name, type and path.
	141
	142	The following two models are used under the hood to represent the metadata:
	143
	144	* [`Entry(name, type, path)`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/entry.rb)
	145	* [`Type(name, slug, count)`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/type.rb)
	146
	147	Each scraper must implement its own `EntriesFilter` by subclassing the [`Docs::EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) class. The base class already implements the `call` method and includes four methods which the subclasses can override:
	148
	149	* `get_name` [String]
	150	The name of the default entry (aka. the page's name).
	151	It is usually guessed from the `slug` (documented above) or by searching the HTML markup.
	152	Default: modified version of `slug` (underscores are replaced with spaces and forward slashes with dots)
	153
	154	* `get_type` [String]
	155	The type of the default entry (aka. the page's type).
	156	Entries without a type can be searched for but won't be listed in the app's sidebar (unless no other entries have a type).
	157	Default: `nil`
	158
	159	* `include_default_entry?` [Boolean]
	160	Whether to include the default entry.
	161	Used when a page consists of multiple entries (returned by `additional_entries`) but doesn't have a name/type of its own, or to remove a page from the index (if it has no additional entries), in which case it won't be copied on the local filesystem and any link to it in the other pages will be broken (as explained on the [Scraper Reference](./scraper-reference.md) page, this is used to keep the `:skip` / `:skip_patterns` options to a maintainable size, or if the page includes links that can't reached from anywhere else).
	162	Default: `true`
	163
	164	* `additional_entries` [Array]
	165	The list of additional entries.
	166	Each entry is represented by an Array of three attributes: its name, fragment identifier, and type. The fragment identifier refers to the `id` attribute of the HTML element (usually a heading) that the entry relates to. It is combined with the page's path to become the entry's path. If absent or `nil`, the page's path is used. If the type is absent or `nil`, the default `type` is used.
	167	Example: `[ ['One'], ['Two', 'id'], ['Three', nil, 'type'] ]` adds three additional entries, the first one named "One" with the default path and type, the second one named "Two" with the URL fragment "#id" and the default type, and the third one named "Three" with the default path and the type "type".
	168	The list is usually constructed by running through the markup. Exceptions can also be hard-coded for specific pages.
	169	Default: `[]`
	170
	171	The following accessors are also available, but must not be overridden:
	172
	173	* `name` [String]
	174	Memoized version of `get_name` (`nil` for the root page).
	175
	176	* `type` [String]
	177	Memoized version of `get_type` (`nil` for the root page).
	178
	179	Notes:
	180
	181	* Leading and trailing whitespace is automatically removed from names and types.
	182	* Names must be unique across the documentation and as short as possible (ideally less than 30 characters). Whenever possible, methods should be differentiated from properties by appending `()`, and instance methods should be differentiated from class methods using the `Class#method` or `object.method` conventions.
	183	* You can call `name` from `get_type` or `type` from `get_name` but doing both will cause a stack overflow (i.e. you can infer the name from the type or the type from the name, but you can't do both at the same time). Don't call `get_name` or `get_type` directly as their value isn't memoized.
	184	* The root page has no name and no type (both are `nil`). `get_name` and `get_type` won't get called with the page (but `additional_entries` will).
	185	* `Docs::EntriesFilter` is an _HTML filter_. It must be added to the scraper's `html_filters` stack.
	186	* Try to document the code as much as possible, particularly the special cases. It'll make updating the documentation easier.
	187
	188	Example:
	189
	190	```ruby
	191	module Docs
	192	class MyScraper
	193	class EntriesFilter < Docs::EntriesFilter
	194	def get_name
	195	node = at_css('h1')
	196	result = node.content.strip
	197	result << ' event' if type == 'Events'
	198	result << '()' if node['class'].try(:include?, 'function')
	199	result
	200	end
	201
	202	def get_type
	203	object, method = *slug.split('/')
	204	method ? object : 'Miscellaneous'
	205	end
	206
	207	def additional_entries
	208	return [] if root_page?
	209
	210	css('h2').map do \|node\|
	211	[node.content, node['id']]
	212	end
	213	end
	214
	215	def include_default_entry?
	216	!at_css('.obsolete')
	217	end
	218	end
	219	end
	220	end
	221	```
	222
	223
	224	return [[Home]]

docs/maintainers.md+102−0

@@ -0,0 +1,102 @@
	1	# Maintainer's Guide
	2
	3	This document is intended for [DevDocs maintainers](#list-of-maintainers).
	4
	5	## Merging pull requests
	6
	7	- PRs should be approved by at least one maintainer before being merged.
	8
	9	- PRs that add or update documentations should always be merged locally, and the app deployed, before the merge is pushed to GitHub.
	10
	11	This workflow is required because there is a dependency between the local and production environments. The `thor docs:download` command downloads documentations from production files uploaded by the `thor docs:upload` command. If a PR adding a new documentation is merged and pushed to GitHub before the files have been uploaded to production, the `thor docs:download` will fail for the new documentation and the docker container will not build properly until the new documentation is deployed to production.
	12
	13	## Updating docs
	14
	15	The process for updating docs is as follow:
	16
	17	- Make version/release changes in the scraper file.
	18	- If needed, update the copyright notice of the documentation in the scraper file (`options[:attribution]`) and the about page (`about_tmpl.coffee`). The copyright notice must be the same as the one on the original documentation.
	19	- Run `thor docs:generate`.
	20	- Make sure the documentation still works well. The `thor docs:generate` command outputs a summary of the changes, which helps identifying issues (e.g. deleted files) and new pages to check out in the app. Verify locally that everything works, especially the files that were created (if any), and that the categorization of entries is still good. Often, updates will require code changes to tweak some new markup in the source website or categorize new entries.
	21	- Commit the changes (protip: use the `thor docs:commit` command documented below).
	22	- Optional: do more updates.
	23	- Run `thor docs:upload` (documented below).
	24	- Push to GitHub to [deploy the app](#deploying-devdocs) and verify that everything works in production.
	25	- Run `thor docs:clean` (documented below).
	26
	27	Note: changes to `public/docs/docs.json` should never be committed. This file reflects which documentations have been downloaded or generated locally, which is always none on a fresh `git clone`.
	28
	29	## Setup requirements
	30
	31	In order to deploy DevDocs, you must:
	32
	33	- be given access to Heroku, [configure the Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli) on your computer, and familiarize yourself with Heroku's UI and CLI, as well as that of New Relic (accessible through [the Heroku dashboard](https://dashboard.heroku.com/apps/devdocs)).
	34
	35	- be given access to DevDocs's [Sentry instance](https://sentry.io/devdocs/devdocs-js/) (for JS error tracking) and familiarize yourself with its UI.
	36
	37	- be provided with DevDocs's S3 credentials, and install (`brew install awscli` on macOS) and [configure](https://docs.aws.amazon.com/cli/latest/reference/configure/) the AWS CLI on your computer. The configuration must add a named profile called "devdocs":
	38	```
	39	aws configure --profile devdocs
	40	```
	41
	42	- be provided with DevDocs's MaxCDN push zone credentials, and add them to your `.bash_profile` as such:
	43	```
	44	export DEVDOCS_DL_USERNAME="username"
	45	export DEVDOCS_DL_PASSWORD="password"
	46	```
	47
	48	## Thor commands
	49
	50	In addition to the [publicly-documented commands](https://github.com/freeCodeCamp/devdocs#available-commands), the following commands are aimed at DevDocs maintainers:
	51
	52	- `thor docs:package`
	53
	54	Generates packages for one or more documentations. Those packages are intended to be uploaded to DevDocs's MaxCDN push zone by maintainers via the `thor docs:upload` command, and downloaded by users via the `thor docs:download` command.
	55
	56	Versions can be specified as such: `thor docs:package rails@5.2 node@10\ LTS`.
	57
	58	Packages can also be automatically generated during the scraping process by passing the `--package` option to `thor docs:generate`.
	59
	60	- `thor docs:upload`
	61
	62	This command does two operations:
	63
	64	1. sync the files for the specified documentations with S3 (used by the Heroku app);
	65	2. upload the documentations' packages to DevDocs's MaxCDN push zone (used by the `thor docs:download` command).
	66
	67	For the command to work, you must have the AWS CLI and MaxCDN credentials configured as indicated above.
	68
	69	Important: the app should always be deployed immediately after this command has finished running. Do not run this command unless you are able and ready to deploy DevDocs.
	70
	71	To upload all documentations that are packaged on your computer, run `thor docs:upload --packaged`.
	72	To test your configuration and the effect of this command without uploading anything, pass the `--dryrun` option.
	73
	74	- `thor docs:commit`
	75
	76	Shortcut command to create a Git commit for a given documentation once it has been updated. Scraper and `assets/` file changes will be committed. The commit message will include the most recent version that the documentation was updated to. If some files were missed by the commit, use `git commit --amend` to add them to the commit. The command may be run before `thor docs:upload` is run, but the commit should not be pushed to GitHub before the files have been uploaded and the app deployed.
	77
	78	- `thor docs:clean`
	79
	80	Shortcut command to delete all package files (once uploaded via `thor docs:upload`, they are not needed anymore).
	81
	82	## Deploying DevDocs
	83
	84	Once docs have been uploaded via `thor docs:upload` (if applicable), you can push to the DevDocs master branch (or merge the PR containing the updates). If the Travis build succeeds, the Heroku application will be deployed automatically.
	85
	86	- If you're deploying documentation updates, verify that the documentations work properly once the deploy is done. Keep in mind that you'll need to wait a few seconds for the service worker to finish caching the new assets. You should see a "DevDocs has been updated" notification appear when the caching is done, after which you need to refresh the page to see the changes.
	87	- If you're deploying frontend changes, monitor [Sentry](https://sentry.io/devdocs/devdocs-js/) for new JS errors once the deploy is done.
	88	- If you're deploying server changes, monitor New Relic (accessible through [the Heroku dashboard](https://dashboard.heroku.com/apps/devdocs)) for Ruby exceptions and throughput or response time changes once the deploy is done.
	89
	90	If any issue arises, run `heroku rollback` to rollback to the previous version of the app (this can also be done via Heroku's UI). Note that this will not revert changes made to documentation files that were uploaded via `thor docs:upload`. Try and fix the issue as quickly as possible, then re-deploy the app. Reach out to other maintainers if you need help.
	91
	92	If this is your first deploy, make sure another maintainer is around to assist.
	93
	94	## List of maintainers
	95
	96	- [Jed Fox](https://github.com/j-f1)
	97	- [Jasper van Merle](https://github.com/jmerle)
	98	- [Ahmad Abdolsaheb](https://github.com/ahmadabdolsaheb)
	99	- [Mrugesh Mohapatra](https://github.com/raisedadead)
	100	- [Thibaut Courouble](https://github.com/thibaut)
	101
	102	Interested in helping maintain DevDocs? Come talk to us on [Gitter](https://gitter.im/FreeCodeCamp/DevDocs) :)

docs/scraper-reference.md+227−0

@@ -0,0 +1,227 @@
	1	Table of contents:
	2
	3	* [Overview](#overview)
	4	* [Configuration](#configuration)
	5	- [Attributes](#attributes)
	6	- [Filter stacks](#filter-stacks)
	7	- [Filter options](#filter-options)
	8
	9	## Overview
	10
	11	Starting from a root URL, scrapers recursively follow links that match a set of rules, passing each valid response through a chain of filters before writing the file on the local filesystem. They also create an index of the pages' metadata (determined by one filter), which is dumped into a JSON file at the end.
	12
	13	Scrapers rely on the following libraries:
	14
	15	* [Typhoeus](https://github.com/typhoeus/typhoeus) for making HTTP requests
	16	* [HTML::Pipeline](https://github.com/jch/html-pipeline) for applying filters
	17	* [Nokogiri](http://nokogiri.org/) for parsing HTML
	18
	19	There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base URL with a local path before reading a file. `FileScraper` uses the placeholder `localhost` base URL by default and includes a filter to remove any URL pointing to it at the end.
	20
	21	To be processed, a response must meet the following requirements:
	22
	23	* 200 status code
	24	* HTML content type
	25	* effective URL (after redirection) contained in the base URL (explained below)
	26
	27	(`FileScraper` only checks if the file exists and is not empty.)
	28
	29	Each URL is requested only once (case-insensitive).
	30
	31	## Configuration
	32
	33	Configuration is done via class attributes and divided into three main categories:
	34
	35	* [Attributes](#attributes) — essential information such as name, version, URL, etc.
	36	* [Filter stacks](#filter-stacks) — the list of filters that will be applied to each page.
	37	* [Filter options](#filter-options) — the options passed to said filters.
	38
	39	Note: scrapers are located in the [`lib/docs/scrapers`](https://github.com/Thibaut/devdocs/tree/master/lib/docs/scrapers/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
	40
	41	### Attributes
	42
	43	* `name` [String]
	44	Must be unique.
	45	Defaults to the class's name.
	46
	47	* `slug` [String]
	48	Must be unique, lowercase, and not include dashes (underscores are ok).
	49	Defaults to `name` lowercased.
	50
	51	* `type` [String] (required, inherited)
	52	Defines the CSS class name (`_[type]`) and custom JavaScript class (`app.views.[Type]Page`) that will be added/loaded on each page. Documentations sharing a similar structure (e.g. generated with the same tool or originating from the same website) should use the same `type` to avoid duplicating the CSS and JS.
	53	Must include lowercase letters only.
	54
	55	* `release` [String] (required)
	56	The version of the software at the time the scraper was last run. This is only informational and doesn't affect the scraper's behavior.
	57
	58	* `base_url` [String] (required in `UrlScraper`)
	59	The documents' location. Only URLs _inside_ the `base_url` will be scraped. "inside" more or less means "starting with" except that `/docs` is outside `/doc` (but `/doc/` is inside).
	60	Defaults to `localhost` in `FileScraper`. _(Note: any iframe, image, or skipped link pointing to localhost will be removed by the `CleanLocalUrls` filter; the value should be overridden if the documents are available online.)_
	61	Unless `root_path` is set, the root/initial URL is equal to `base_url`.
	62
	63	* `root_path` [String] (inherited)
	64	The path from the `base_url` of the root URL.
	65
	66	* `initial_paths` [Array] (inherited)
	67	A list of paths (from the `base_url`) to add to the initial queue. Useful for scraping isolated documents.
	68	Defaults to `[]`. _(Note: the `root_path` is added to the array at runtime.)_
	69
	70	* `dir` [String] (required, `FileScraper` only)
	71	The absolute path where the files are located on the local filesystem.
	72	_Note: `FileScraper` works exactly like `UrlScraper` (manipulating the same kind of URLs) except that it substitutes `base_url` with `dir` in order to read files instead of making HTTP requests._
	73
	74	* `params` [Hash] (inherited, `UrlScraper` only)
	75	Query string parameters to append to every URL. (e.g. `{ format: 'raw' }` → `?format=raw`)
	76	Defaults to `{}`.
	77
	78	* `abstract` [Boolean]
	79	Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/scrapers/mdn/mdn.rb) class).
	80	Defaults to `false`.
	81
	82	### Filter stacks
	83
	84	Each scraper has two [filter](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter.rb) [stacks](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter_stack.rb): `html_filters` and `text_filters`. They are combined into a pipeline (using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library) which causes each filter to hand its output to the next filter's input.
	85
	86	HTML filters are executed first and manipulate a parsed version of the document (a [Nokogiri](http://nokogiri.org/Nokogiri/XML/Node.html) node object), whereas text filters manipulate the document as a string. This separation avoids parsing the document multiple times.
	87
	88	Filter stacks are like sorted sets. They can modified using the following methods:
	89
	90	```ruby
	91	push(*names) # append one or more filters at the end
	92	insert_before(index, *names) # insert one filter before another (index can be a name)
	93	insert_after(index, *names) # insert one filter after another (index can be a name)
	94	replace(index, name) # replace one filter with another (index can be a name)
	95	```
	96
	97	"names" are `require` paths relative to `Docs` (e.g. `jquery/clean_html` → `Docs::Jquery::CleanHtml`).
	98
	99	Default `html_filters`:
	100
	101	* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
	102	* [`CleanHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
	103	* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
	104	* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
	105	* [`NormalizePathsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
	106	* [`CleanLocalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
	107
	108	Default `text_filters`:
	109
	110	* [`InnerHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
	111	* [`CleanTextFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
	112	* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
	113
	114	Additionally:
	115
	116	* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb) is a core HTML filter, disabled by default, which prepends the document with a title (`<h1>`).
	117	* [`EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) is an abstract HTML filter that each scraper must implement and responsible for extracting the page's metadata.
	118
	119	### Filter options
	120
	121	The filter options are stored in the `options` Hash. The Hash is inheritable (a recursive copy) and empty by default.
	122
	123	More information about how filters work is available on the [Filter Reference](./filter-reference.md) page.
	124
	125	* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb)
	126
	127	- `:container` [String or Proc]
	128	A CSS selector of the container element. Everything outside of it will be removed and become unavailable to the other filters. If more than one element match the selector, the first one inside the DOM is used. If no elements match the selector, an error is raised.
	129	If the value is a Proc, it is called for each page with the filter instance as argument, and should return a selector or `nil`.
	130	The default container is the `<body>` element.
	131	_Note: links outside of the container element will not be followed by the scraper. To remove links that should be followed, use a [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter later in the stack._
	132
	133	* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb)
	134	The following options are used to modify URLs in the pages. They are useful to remove duplicates (when the same page is accessible from multiple URLs) and fix websites that have a bunch of redirections in place (when URLs that should be scraped, aren't, because they are behind a redirection which is outside of the `base_url` — see the MDN scrapers for examples of this).
	135
	136	- `:replace_urls` [Hash]
	137	Replaces all instances of a URL with another.
	138	Format: `{ 'original_url' => 'new_url' }`
	139	- `:replace_paths` [Hash]
	140	Replaces all instances of a sub-path (path from the `base_url`) with another.
	141	Format: `{ 'original_path' => 'new_path' }`
	142	- `:fix_urls` [Proc]
	143	Called with each URL. If the returned value is `nil`, the URL isn't modified. Otherwise the returned value is used as replacement.
	144
	145	_Note: before these rules are applied, all URLs are converted to their fully qualified counterpart (http://...)._
	146
	147	* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb)
	148
	149	Internal URLs are the ones _inside_ the scraper's `base_url` ("inside" more or less means "starting with", except that `/docs` is outside `/doc`). They will be scraped unless excluded by one of the following rules. All internal URLs are converted to relative URLs inside the pages.
	150
	151	- `:skip_links` [Boolean or Proc]
	152	If `false`, does not convert or follow any internal URL (creating a single-page documentation).
	153	If the value is a Proc, it is called for each page with the filter instance as argument.
	154	- `:follow_links` [Proc]
	155	Called for page with the filter instance as argument. If the returned value is `false`, does not add internal URLs to the queue.
	156	- `:trailing_slash` [Boolean]
	157	If `true`, adds a trailing slash to all internal URLs. If `false`, removes it.
	158	This is another option used to remove duplicate pages.
	159	- `:skip` [Array]
	160	Ignores internal URLs whose sub-paths (path from the `base_url`) are in the Array (case-insensitive).
	161	- `:skip_patterns` [Array]
	162	Ignores internal URLs whose sub-paths match any Regexp in the Array.
	163	- `:only` [Array]
	164	Ignores internal URLs whose sub-paths aren't in the Array (case-insensitive) and don't match any Regexp in `:only_patterns`.
	165	- `:only_patterns` [Array]
	166	Ignores internal URLs whose sub-paths don't match any Regexp in the Array and aren't in `:only`.
	167
	168	If the scraper has a `root_path`, the empty and `/` paths are automatically skipped.
	169	If `:only` or `:only_patterns` is set, the root path is automatically added to `:only`.
	170
	171	_Note: pages can be excluded from the index based on their content using the [`Entries`](./filter-reference.md#entriesfilter) filter. However, their URLs will still be converted to relative in the other pages and trying to open them will return a 404 error. Although not ideal, this is often better than having to maintain a long list of `:skip` URLs._
	172
	173	* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb)
	174
	175	- `:attribution` [String] (required)
	176	An HTML string with the copyright and license information. See the other scrapers for examples.
	177
	178	* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb)
	179
	180	- `:title` [String or Boolean or Proc]
	181	Unless the value is `false`, adds a title to every page.
	182	If the value is `nil`, the title is the name of the page as determined by the [`Entries`](./filter-reference.md#entriesfilter) filter. Otherwise the title is the String or the value returned by the Proc (called for each page, with the filter instance as argument). If the Proc returns `nil` or `false`, no title is added.
	183	- `:root_title` [String or Boolean]
	184	Overrides the `:title` option for the root page only.
	185
	186	_Note: this filter is disabled by default._
	187
	188	## Keeping scrapers up-to-date
	189
	190	In order to keep scrapers up-to-date the `get_latest_version(opts)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.
	191
	192	To make life easier, there are a few utility methods that you can use in `get_latest_version`:
	193	* `fetch(url, opts)`
	194
	195	Makes a GET request to the url and returns the response body.
	196
	197	Example: [lib/docs/scrapers/bash.rb](../lib/docs/scrapers/bash.rb)
	198	* `fetch_doc(url, opts)`
	199
	200	Makes a GET request to the url and returns the HTML body converted to a Nokogiri document.
	201
	202	Example: [lib/docs/scrapers/git.rb](../lib/docs/scrapers/git.rb)
	203	* `fetch_json(url, opts)`
	204
	205	Makes a GET request to the url and returns the JSON body converted to a dictionary.
	206
	207	Example: [lib/docs/scrapers/mdn/mdn.rb](../lib/docs/scrapers/mdn/mdn.rb)
	208	* `get_npm_version(package, opts)`
	209
	210	Returns the latest version of the given npm package.
	211
	212	Example: [lib/docs/scrapers/bower.rb](../lib/docs/scrapers/bower.rb)
	213	* `get_latest_github_release(owner, repo, opts)`
	214
	215	Returns the tag name of the latest GitHub release of the given repository. If the tag name is preceded by a "v", the "v" will be removed.
	216
	217	Example: [lib/docs/scrapers/jsdoc.rb](../lib/docs/scrapers/jsdoc.rb)
	218	* `get_github_tags(owner, repo, opts)`
	219
	220	Returns the list of tags on the given repository ([format](https://developer.github.com/v3/repos/#list-tags)).
	221
	222	Example: [lib/docs/scrapers/liquid.rb](../lib/docs/scrapers/liquid.rb)
	223	* `get_github_file_contents(owner, repo, path, opts)`
	224
	225	Returns the contents of the requested file in the default branch of the given repository.
	226
	227	Example: [lib/docs/scrapers/minitest.rb](../lib/docs/scrapers/minitest.rb)

lib/app.rb+43−25

   Rack::Mime::MIME_TYPES['.webapp'] = 'application/x-web-app-manifest+json'
   configure do
-    use Rack::SslEnforcer, only_environments: ['production', 'test'], hsts: false, force_secure_cookies: false
+    use Rack::SslEnforcer, only_environments: ['production', 'test'], hsts: true, force_secure_cookies: false
     set :sentry_dsn, ENV['SENTRY_DSN']
     set :protection, except: [:frame_options, :xss_header]
     set :csp, false
+    require 'docs'
+    Docs.generate_manifest
     Dir[docs_path, root.join(assets_prefix, '*/')].each do |path|
       sprockets.append_path(path)
     end
   end
   configure :test, :development do
+    require 'thor'
+    load 'tasks/sprites.thor'
+    SpritesCLI.new.invoke(:generate)
     require 'active_support/per_thread_registry'
     require 'active_support/cache'
     sprockets.cache = ActiveSupport::Cache.lookup_store :file_store, root.join('tmp', 'cache', 'assets', environment.to_s)
       request.query_string.empty? ? nil : "?#{request.query_string}"
     end
-    def manifest_asset_urls
+    def service_worker_asset_urls
-      @@manifest_asset_urls ||= [
+      @@service_worker_asset_urls ||= [
         javascript_path('application', asset_host: false),
         stylesheet_path('application'),
-        image_path('docs-1.png'),
+        image_path('sprites/docs.png'),
-        image_path('docs-1@2x.png'),
+        image_path('sprites/docs@2x.png'),
-        image_path('docs-2.png'),
+        asset_path('docs.js'),
-        image_path('docs-2@2x.png'),
+        App.production? ? nil : javascript_path('debug'),
-        asset_path('docs.js')
+      ].compact
-      ]
     end
-    def app_size
+    # Returns a cache name for the service worker to use which changes if any of the assets changes
-      @app_size ||= memoized_cookies['size'].nil? ? '20rem' : "#{memoized_cookies['size']}px"
+    # When a manifest exist, this name is only created once based on the asset manifest because it never changes without a server restart
-    end
+    # If a manifest does not exist, it is created every time this method is called because the assets can change while the server is running
+    def service_worker_cache_name
+      if File.exist?(App.assets_manifest_path)
+        if defined?(@@service_worker_cache_name)
+          return @@service_worker_cache_name
+        end
-    def app_layout
+        digest = Sprockets::Manifest
-      memoized_cookies['layout']
+                   .new(nil, App.assets_manifest_path)
-    end
+                   .files
+                   .values
+                   .map {|file| file["digest"]}
+                   .join
-    def app_theme
+        return @@service_worker_cache_name ||= Digest::MD5.hexdigest(digest)
-      @app_theme ||= memoized_cookies['dark'].nil? ? 'default' : 'dark'
+      else
-    end
+        paths = App.sprockets
+                  .each_file
+                  .to_a
+                  .reject {|file| file.start_with?(App.docs_path)}
-    def dark_theme?
+        return App.sprockets.pack_hexdigest(App.sprockets.files_digest(paths))
-      app_theme == 'dark'
+      end
     end
-    def redirect_via_js(path) # courtesy of HTML5 App Cache
+    def redirect_via_js(path)
       response.set_cookie :initial_path, value: path, expires: Time.now + 15, path: '/'
       redirect '/', 302
     end
     end
   end
-  get '/manifest.appcache' do
+  get '/service-worker.js' do
-    content_type 'text/cache-manifest'
+    content_type 'application/javascript'
     expires 0, :'no-cache'
-    erb :manifest
+    erb :'service-worker.js'
   end
   get '/' do
     return redirect "/#q=#{params[:q]}" if params[:q]
-    return redirect '/' unless request.query_string.empty? # courtesy of HTML5 App Cache
+    return redirect '/' unless request.query_string.empty?
     response.headers['Content-Security-Policy'] = settings.csp if settings.csp
     erb :index
   end

lib/docs.rb+17−0

@@ -29,6 +29,7 @@ module Docs · 28 unmodified lines
29	self.rescue_errors = false	29	self.rescue_errors = false
30		30
31	class DocNotFound < NameError; end	31	class DocNotFound < NameError; end
		32	class SetupError < StandardError; end
32		33
33	def self.all	34	def self.all
34	Dir["#{root_path}/docs/scrapers/*/.rb"].	35	Dir["#{root_path}/docs/scrapers/*/.rb"].
@@ -74,6 +75,22 @@ module Docs · 39 unmodified lines
74	end	75	end
75	end	76	end
76		77
		78	def self.find_by_slug(slug, version = nil)
		79	doc = all.find { \|klass\| klass.slug == slug }
		80
		81	unless doc
		82	raise DocNotFound.new(%(could not find doc with "#{slug}"), slug)
		83	end
		84
		85	if version.present?
		86	version = doc.versions.find { \|klass\| klass.version == version \|\| klass.version_slug == version }
		87	raise DocNotFound.new(%(could not find version "#{version}" for doc "#{doc.name}"), doc.name) unless version
		88	doc = version
		89	end
		90
		91	doc
		92	end
		93
77	def self.generate_page(name, version, page_id)	94	def self.generate_page(name, version, page_id)
78	find(name, version).store_page(store, page_id)	95	find(name, version).store_page(store, page_id)
79	end	96	end

lib/docs/core/doc.rb+105−1

@@ -95,6 +95,9 @@ module Docs · 94 unmodified lines
95	false	95	false
96	end	96	end
97	end	97	end
		98	rescue Docs::SetupError => error
		99	puts "ERROR: #{error.message}"
		100	false
98	end	101	end
99		102
100	def store_pages(store)	103	def store_pages(store)
@@ -118,6 +121,9 @@ module Docs · 17 unmodified lines
118	false	121	false
119	end	122	end
120	end	123	end
		124	rescue Docs::SetupError => error
		125	puts "ERROR: #{error.message}"
		126	false
121	end	127	end
122		128
123	private	129	private
@@ -146,7 +152,6 @@ module Docs · 22 unmodified lines
146	end	152	end
147	end	153	end
148		154
149
150	def initialize	155	def initialize
151	raise NotImplementedError, "#{self.class} is an abstract class and cannot be instantiated." if self.class.abstract	156	raise NotImplementedError, "#{self.class} is an abstract class and cannot be instantiated." if self.class.abstract
152	end	157	end
@@ -158,5 +163,104 @@ module Docs · 5 unmodified lines
158	def build_pages(&block)	163	def build_pages(&block)
159	raise NotImplementedError	164	raise NotImplementedError
160	end	165	end
		166
		167	def get_scraper_version(opts)
		168	if self.class.method_defined?(:options) and !options[:release].nil?
		169	options[:release]
		170	else
		171	# If options[:release] does not exist, we return the Epoch timestamp of when the doc was last modified in DevDocs production
		172	json = fetch_json('https://devdocs.io/docs.json', opts)
		173	items = json.select {\|item\| item['name'] == self.class.name}
		174	items = items.map {\|item\| item['mtime']}
		175	items.max
		176	end
		177	end
		178
		179	# Should return the latest version of this documentation
		180	# If options[:release] is defined, it should be in the same format
		181	# If options[:release] is not defined, it should return the Epoch timestamp of when the documentation was last updated
		182	# If the docs will never change, simply return '1.0.0'
		183	def get_latest_version(opts)
		184	raise NotImplementedError
		185	end
		186
		187	# Returns whether or not this scraper is outdated.
		188	#
		189	# The default implementation assumes the documentation uses a semver(-like) approach when it comes to versions.
		190	# Patch updates are ignored because there are usually little to no documentation changes in bug-fix-only releases.
		191	#
		192	# Scrapers of documentations that do not use this versioning approach should override this method.
		193	#
		194	# Examples of the default implementation:
		195	# 1 -> 2 = outdated
		196	# 1.1 -> 1.2 = outdated
		197	# 1.1.1 -> 1.1.2 = not outdated
		198	def is_outdated(scraper_version, latest_version)
		199	scraper_parts = scraper_version.to_s.split(/\./).map(&:to_i)
		200	latest_parts = latest_version.to_s.split(/\./).map(&:to_i)
		201
		202	# Only check the first two parts, the third part is for patch updates
		203	[0, 1].each do \|i\|
		204	break if i >= scraper_parts.length or i >= latest_parts.length
		205	return true if latest_parts[i] > scraper_parts[i]
		206	return false if latest_parts[i] < scraper_parts[i]
		207	end
		208
		209	false
		210	end
		211
		212	private
		213
		214	#
		215	# Utility methods for get_latest_version
		216	#
		217
		218	def fetch(url, opts)
		219	headers = {}
		220
		221	if opts.key?(:github_token) and url.start_with?('https://api.github.com/')
		222	headers['Authorization'] = "token #{opts[:github_token]}"
		223	end
		224
		225	opts[:logger].debug("Fetching #{url}")
		226	response = Request.run(url, { connecttimeout: 15, headers: headers })
		227
		228	if response.success?
		229	response.body
		230	else
		231	reason = response.timed_out? ? "Timed out while connecting to #{url}" : "Couldn't fetch #{url} (response code #{response.code})"
		232	opts[:logger].error(reason)
		233	raise reason
		234	end
		235	end
		236
		237	def fetch_doc(url, opts)
		238	body = fetch(url, opts)
		239	Nokogiri::HTML.parse(body, nil, 'UTF-8')
		240	end
		241
		242	def fetch_json(url, opts)
		243	JSON.parse fetch(url, opts)
		244	end
		245
		246	def get_npm_version(package, opts)
		247	json = fetch_json("https://registry.npmjs.com/#{package}", opts)
		248	json['dist-tags']['latest']
		249	end
		250
		251	def get_latest_github_release(owner, repo, opts)
		252	release = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", opts)
		253	tag_name = release['tag_name']
		254	tag_name.start_with?('v') ? tag_name[1..-1] : tag_name
		255	end
		256
		257	def get_github_tags(owner, repo, opts)
		258	fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", opts)
		259	end
		260
		261	def get_github_file_contents(owner, repo, path, opts)
		262	json = fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", opts)
		263	Base64.decode64(json['content'])
		264	end
161	end	265	end
162	end	266	end

lib/docs/core/requester.rb+8−4

 module Docs
   class Requester < Typhoeus::Hydra
+    include Instrumentable
     attr_reader :request_options
     def self.run(urls, options = {}, &block)
     def initialize(options = {})
       @request_options = options.extract!(:request_options)[:request_options].try(:dup) || {}
       options[:max_concurrency] ||= 20
-      options[:pipelining] = false
+      options[:pipelining] = 0
       super
     end
     end
     def handle_response(response)
-      on_response.each do |callback|
+      instrument 'handle_response.requester', url: response.url do
-        result = callback.call(response)
+        on_response.each do |callback|
-        result.each { |url| request(url) } if result.is_a? Array
+          result = callback.call(response)
+          result.each { |url| request(url) } if result.is_a?(Array)
+        end
       end
     end
   end

lib/docs/core/scrapers/file_scraper.rb+17−6

 module Docs
   class FileScraper < Scraper
+    SOURCE_DIRECTORY = File.expand_path '../../../../../docs', __FILE__
     Response = Struct.new :body, :url
     class << self
-      attr_accessor :dir
       def inherited(subclass)
         super
         subclass.base_url = base_url
-        subclass.dir = dir
       end
     end
     html_filters.push 'clean_local_urls'
+    def source_directory
+      @source_directory ||= File.join(SOURCE_DIRECTORY, self.class.path)
+    end
     private
+    def assert_source_directory_exists
+      unless Dir.exists?(source_directory)
+        raise SetupError, "The #{self.class.name} scraper requires the original documentation files to be stored in the \"#{source_directory}\" directory."
+      end
+    end
     def request_one(url)
-      Response.new read_file(file_path_for(url)), URL.parse(url)
+      assert_source_directory_exists
+      Response.new read_file(File.join(source_directory, url_to_path(url))), URL.parse(url)
     end
     def request_all(urls)
+      assert_source_directory_exists
       queue = [urls].flatten
       until queue.empty?
         result = yield request_one(queue.shift)
       response.body.present?
     end
-    def file_path_for(url)
+    def url_to_path(url)
-      File.join self.class.dir, url.remove(base_url.to_s)
+      url.remove(base_url.to_s)
     end
     def read_file(path)

lib/docs/core/url.rb+1−1

@@ -72,7 +72,7 @@ module Docs · 71 unmodified lines
72		72
73	if base == dest	73	if base == dest
74	''	74	''
75	elsif dest.start_with? File.join(base, '')	75	elsif dest.start_with?(::File.join(base, ''))
76	url.path[(path.length)..-1]	76	url.path[(path.length)..-1]
77	end	77	end
78	end	78	end

lib/docs/filters/cmake/entries.rb+1−0

@@ -20,6 +20,7 @@ module Docs · 19 unmodified lines
20	'prop_sf' => 'Properties: Source Files',	20	'prop_sf' => 'Properties: Source Files',
21	'prop_test' => 'Properties: Tests',	21	'prop_test' => 'Properties: Tests',
22	'prop_tgt' => 'Properties: Targets',	22	'prop_tgt' => 'Properties: Targets',
		23	'envvar' => 'Environment Variables',
23	'variable' => 'Variables' }	24	'variable' => 'Variables' }
24		25
25	def get_name	26	def get_name

lib/docs/filters/composer/clean_html.rb+35−0

@@ -0,0 +1,35 @@
	1	module Docs
	2	class Composer
	3	class CleanHtmlFilter < Filter
	4	def call
	5	# Remove unneeded elements
	6	css('#searchbar, .toc, .fork-and-edit, .anchor').remove
	7
	8	# Fix the home page titles
	9	if subpath == ''
	10	css('h1').each do \|node\|
	11	node.name = 'h2'
	12	end
	13
	14	# Add a main title before the first subtitle
	15	at_css('h2').before('<h1>Composer</h1>')
	16	end
	17
	18	# Code blocks
	19	css('pre').each do \|node\|
	20	code = node.at_css('code[class]')
	21
	22	unless code.nil?
	23	node['data-language'] = 'javascript' if code['class'].include?('javascript')
	24	node['data-language'] = 'php' if code['class'].include?('php')
	25	end
	26
	27	node.content = node.content.strip
	28	node.remove_attribute('class')
	29	end
	30
	31	doc
	32	end
	33	end
	34	end
	35	end

lib/docs/filters/composer/entries.rb+36−0

@@ -0,0 +1,36 @@
	1	module Docs
	2	class Composer
	3	class EntriesFilter < Docs::EntriesFilter
	4	def get_name
	5	title = at_css('h1').content
	6	title = "#{Integer(subpath[1]) + 1}. #{title}" if type == 'Book'
	7	title
	8	end
	9
	10	def get_type
	11	return 'Articles' if subpath.start_with?('articles/')
	12	'Book'
	13	end
	14
	15	def additional_entries
	16	entries = []
	17
	18	if subpath == '04-schema.md' # JSON Schema
	19	css('h3').each do \|node\|
	20	name = node.content.strip
	21	name.remove!(' (root-only)')
	22	entries << [name, node['id'], 'JSON Schema']
	23	end
	24	end
	25
	26	if subpath == '06-config.md' # Composer config
	27	css('h2').each do \|node\|
	28	entries << [node.content.strip, node['id'], 'Configuration Options']
	29	end
	30	end
	31
	32	entries
	33	end
	34	end
	35	end
	36	end

lib/docs/filters/cpp/entries.rb+2−0

           'Keywords'
         elsif subpath.start_with?('experimental')
           'Experimental libraries'
+        elsif subpath.start_with?('language/')
+          'Language'
         elsif type = at_css('.t-navbar > div:nth-child(4) > :first-child').try(:content)
           type.strip!
           type.remove! ' library'

lib/docs/filters/css/entries.rb+17−10

         'CSS_Background_and_Borders' => 'Backgrounds & Borders',
         'CSS_Columns' => 'Multi-column Layout',
         'CSS_Flexible_Box_Layout' => 'Flexible Box Layout',
+        'CSS_Fonts' => 'Fonts',
         'CSS_Grid_Layout' => 'Grid Layout',
         'CSS_Images' => 'Images',
         'CSS_Lists_and_Counters' => 'Lists',
         'CSS_Transforms' => 'Transforms',
         'Media_Queries' => 'Media Queries',
+        'filter-function' => 'Filter Effects',
         'transform-function' => 'Transforms',
         '@media' => 'Media Queries',
+        'overscroll' => 'Overscroll',
         'text-size-adjust' => 'Miscellaneous',
         'resolved_value' => 'Miscellaneous',
         'touch-action' => 'Miscellaneous',
       end
       def get_type
-        if slug.include?('-webkit') || slug.include?('-moz')
+        if slug.include?('-webkit') || slug.include?('-moz') || slug.include?('-ms')
           'Extensions'
         elsif type = TYPE_BY_PATH[slug.split('/').first]
           type
           'Pseudo-Elements'
         elsif name.start_with?(':')
           'Selectors'
+        elsif name.start_with?('display-')
+          'Display'
         else
           'Miscellaneous'
         end
       end
       STATUSES = {
-        'spec-Living' => 0,
+        'spec-Living'   => 0,
-        'spec-REC'    => 1,
+        'spec-REC'      => 1,
-        'spec-CR'     => 2,
+        'spec-CR'       => 2,
-        'spec-PR'     => 3,
+        'spec-PR'       => 3,
-        'spec-LC'     => 4,
+        'spec-LC'       => 4,
-        'spec-WD'     => 5,
+        'spec-WD'       => 5,
-        'spec-ED'     => 6
+        'spec-ED'       => 6,
+        'spec-Obsolete' => 7
       }
       PRIORITY_STATUSES = %w(spec-REC spec-CR)
       def get_spec
         return unless table = at_css('#Specifications + table') || css('.standard-table').last
         specs = table.css('tbody tr').to_a
         # [link, span]
         specs.map!     { |node| [node.at_css('> td:nth-child(1) > a'), node.at_css('> td:nth-child(2) > span')] }
         'shape' => [
           %w(rect() Syntax) ],
         'timing-function' => [
-          %w(cubic-bezier() The_cubic-bezier()_class_of_timing-functions),
+          %w(cubic-bezier()),
-          %w(steps() The_steps()_class_of_timing-functions),
+          %w(steps()),
           %w(linear linear),
           %w(ease ease),
           %w(ease-in ease-in),

lib/docs/filters/cypress/clean_html.rb+26−0

@@ -0,0 +1,26 @@
	1	# frozen_string_literal: true
	2
	3	module Docs
	4	class Cypress
	5	class CleanHtmlFilter < Filter
	6	def call
	7	article_div = at_css('#article > div')
	8	@doc = article_div unless article_div.nil?
	9
	10	header = at_css('h1.article-title')
	11	doc.prepend_child(header) unless header.nil?
	12
	13	css('.article-edit-link').remove
	14	css('.article-footer').remove
	15	css('.article-footer-updated').remove
	16
	17	css('pre').each do \|node\|
	18	node.content = node.content
	19	node['data-language'] = 'javascript'
	20	end
	21
	22	doc
	23	end
	24	end
	25	end
	26	end

lib/docs/filters/cypress/entries.rb+34−0

@@ -0,0 +1,34 @@
	1	# frozen_string_literal: true
	2
	3	module Docs
	4	class Cypress
	5	class EntriesFilter < Docs::EntriesFilter
	6	SECTIONS = %w[
	7	commands
	8	core-concepts
	9	cypress-api
	10	events
	11	getting-started
	12	guides
	13	overview
	14	plugins
	15	references
	16	utilities
	17	].freeze
	18
	19	def get_name
	20	at_css('h1.article-title').content.strip
	21	end
	22
	23	def get_type
	24	path = context[:url].path
	25
	26	SECTIONS.each do \|section\|
	27	if path.match?("/#{section}/")
	28	return section.split('-').map(&:capitalize).join(' ')
	29	end
	30	end
	31	end
	32	end
	33	end
	34	end

lib/docs/filters/dom/clean_html.rb+1−1

@@ -30,7 +30,7 @@ module Docs · 29 unmodified lines
30	end	30	end
31		31
32	# Remove <div> wrapping .overheadIndicator	32	# Remove <div> wrapping .overheadIndicator
33	css('div > .overheadIndicator:first-child:last-child').each do \|node\|	33	css('div > .overheadIndicator:first-child:last-child', 'div > .blockIndicator:first-child:last-child').each do \|node\|
34	node.parent.replace(node)	34	node.parent.replace(node)
35	end	35	end
36		36

lib/docs/filters/dom/entries.rb+12−2

         'EXT_'                => 'WebGL',
         'OES_'                => 'WebGL',
         'WEBGL_'              => 'WebGL',
+        'Sensor API'          => 'Sensors',
         'Ambient Light'       => 'Ambient Light',
         'Audio'               => 'Audio',
         'Battery Status'      => 'Battery Status',
         'Encrypted Media Extensions' => 'Encrypted Media',
         'Fetch'               => 'Fetch',
         'File API'            => 'File',
+        'Fullscreen'          => 'Fullscreen',
         'Geolocation'         => 'Geolocation',
         'Geometry'            => 'Geometry',
         'High Resolution Time' => 'Performance',
         'Intersection'        => 'Intersection Observer',
+        'Keyboard'            => 'Keyboard',
         'Media Capabilities'  => 'Media',
         'Media Capture'       => 'Media',
         'Media Session'       => 'Media',
         'MIDI'                => 'Audio',
         'Navigation Timing'   => 'Performance',
         'Network Information' => 'Network Information',
+        'Orientation Sensor'  => 'Sensors',
         'Payment'             => 'Payments',
         'Performance Timeline' => 'Performance',
         'Pointer Events'      => 'Pointer Events',
         'Web App Manifest'    => 'Web App Manifest',
         'Budget'              => 'Budget',
         'Web Authentication'  => 'Authentication',
+        'Web Locks'           => 'Locks',
         'Web Workers'         => 'Web Workers',
         'WebGL'               => 'WebGL',
         'WebRTC'              => 'WebRTC',
         'WebUSB'              => 'WebUSB',
-        'WebVR'               => 'WebVR' }
+        'WebVR'               => 'WebVR',
+        'WebVTT'              => 'WebVTT' }
       TYPE_BY_NAME_STARTS_WITH = {
         'AbortController'     => 'Fetch',
         'Fetch'               => 'Fetch',
         'File'                => 'File',
         'GlobalEventHandlers' => 'GlobalEventHandlers',
+        'HMDVR'               => 'WebVR',
         'history'             => 'History',
         'HTML Drag'           => 'Drag & Drop',
         'HTML'                => 'Elements',
         'IDB'                 => 'IndexedDB',
+        'Keyboard'            => 'Keyboard',
         'location'            => 'Location',
         'navigator'           => 'Navigator',
         'MediaKeySession'     => 'Encrypted Media',
         'StyleSheet'          => 'CSS',
         'Stylesheet'          => 'CSS',
         'SVG'                 => 'SVG',
+        'TextTrack'           => 'WebVTT',
         'TimeRanges'          => 'Media',
         'timing'              => 'Performance',
         'Timing'              => 'Performance',
         'timing'        => 'Performance',
         'Timing'        => 'Performance',
         'udio'          => 'Audio',
+        'VRDevice'      => 'WebVR',
         'WebGL'         => 'WebGL',
         'WEBGL'         => 'WebGL',
         'WebRTC'        => 'WebRTC',
       def include_default_entry?
         return true if type == 'Console'
-        return true unless node = doc.at_css('.overheadIndicator')
+        return true unless node = doc.at_css('.overheadIndicator, .blockIndicator')
         node = node.parent while node.parent != doc
         return true if node.previous_element.try(:name).in?(%w(h2 h3))
         content = node.content

lib/docs/filters/elixir/clean_html.rb+5−0

           node.parent.after(node)
         end
+        css('.signature').each do |node|
+          non_text_children = node.xpath('node()[not(self::text())]')
+          non_text_children.to_a.reverse.each { |child| node.parent.add_next_sibling(child) }
+        end
         css('pre').each do |node|
           node['data-language'] = 'elixir'
           node.content = node.content

lib/docs/filters/go/clean_html.rb+1−0

@@ -17,6 +17,7 @@ module Docs · 16 unmodified lines
17	css('#plusone', '#nav', '.pkgGopher', '#footer', '.collapsed', '.permalink', '#pkg-callgraph').remove	17	css('#plusone', '#nav', '.pkgGopher', '#footer', '.collapsed', '.permalink', '#pkg-callgraph').remove
18		18
19	css('span[style]', '.toggleVisible', '.expanded', 'div.toggle').each do \|node\|	19	css('span[style]', '.toggleVisible', '.expanded', 'div.toggle').each do \|node\|
		20	node.first_element_child['id'] = node['id'] if node['id']
20	node.before(node.children).remove	21	node.before(node.children).remove
21	end	22	end
22		23

lib/docs/filters/homebrew/clean_html.rb+5−0

       def call
         css('hr')
+        if at_css('h1').nil?
+          title = current_url.normalized_path[1..-1].gsub(/-/, ' ')
+          doc.children.before("<h1>#{title}</h1>")
+        end
         css('div.highlighter-rouge').each do |node|
           lang = node['class'][/language-(\w+)/, 1]
           node['data-language'] = lang if lang

lib/docs/filters/homebrew/entries.rb+3−1

   class Homebrew
     class EntriesFilter < Docs::EntriesFilter
       def get_name
-        name = at_css('h1').content.strip
+        header = at_css('h1')
+        name = header.nil? ? current_url.normalized_path[1..-1].gsub(/-/, ' ') : header.content.strip
         name.remove! %r{\(.*}
         name
       end
         Python-for-Formula-Authors
         Migrating-A-Formula-To-A-Tap
         Rename-A-Formula
+        Building-Against-Non-Homebrew-Dependencies
         How-to-Create-and-Maintain-a-Tap
         Brew-Test-Bot
         Prose-Style-Guidelines)

lib/docs/filters/html/entries.rb+1−1

@@ -28,7 +28,7 @@ module Docs · 27 unmodified lines
28		28
29	def include_default_entry?	29	def include_default_entry?
30	return false if %w(Element/Heading_Elements).include?(slug)	30	return false if %w(Element/Heading_Elements).include?(slug)
31	(node = doc.at_css '.overheadIndicator').nil? \|\| node.content.exclude?('not on a standards track')	31	(node = doc.at_css '.overheadIndicator, .blockIndicator').nil? \|\| node.content.exclude?('not on a standards track')
32	end	32	end
33		33
34	def additional_entries	34	def additional_entries

lib/docs/filters/javascript/clean_html.rb+2−2

@@ -11,12 +11,12 @@ module Docs · 10 unmodified lines
11		11
12	def other	12	def other
13	# Remove "style" attribute	13	# Remove "style" attribute
14	css('.inheritsbox', '.overheadIndicator').each do \|node\|	14	css('.inheritsbox', '.overheadIndicator', '.blockIndicator').each do \|node\|
15	node.remove_attribute 'style'	15	node.remove_attribute 'style'
16	end	16	end
17		17
18	# Remove <div> wrapping .overheadIndicator	18	# Remove <div> wrapping .overheadIndicator
19	css('div > .overheadIndicator:first-child:last-child').each do \|node\|	19	css('div > .overheadIndicator:first-child:last-child', 'div > .blockIndicator:first-child:last-child').each do \|node\|
20	node.parent.replace(node)	20	node.parent.replace(node)
21	end	21	end
22	end	22	end

lib/docs/filters/javascript/entries.rb+1−1

@@ -88,7 +88,7 @@ module Docs · 87 unmodified lines
88	end	88	end
89		89
90	def include_default_entry?	90	def include_default_entry?
91	node = doc.at_css '.overheadIndicator, .warning'	91	node = doc.at_css '.blockIndicator, .warning'
92		92
93	# Can't use :first-child because #doc is a DocumentFragment	93	# Can't use :first-child because #doc is a DocumentFragment
94	return true unless node && node.parent == doc && !node.previous_element	94	return true unless node && node.parent == doc && !node.previous_element

lib/docs/filters/kotlin/clean_html.rb+7−0

@@ -46,6 +46,13 @@ module Docs · 45 unmodified lines
46	parent.content = parent.content	46	parent.content = parent.content
47	parent['data-language'] = 'kotlin'	47	parent['data-language'] = 'kotlin'
48	end	48	end
		49
		50	css('.tags').each do \|wrapper\|
		51	platforms = wrapper.css('.platform:not(.tag-value-Common)').to_a
		52	platforms = platforms.map { \|node\| "#{node.content} (#{node['data-tag-version']})" }
		53	platforms = "<b>Platform and version requirements:</b> #{platforms.join ", "}"
		54	wrapper.replace(platforms)
		55	end
49	end	56	end
50	end	57	end
51	end	58	end

lib/docs/filters/kotlin/entries.rb+3−1

         if subpath.start_with?('api')
           breadcrumbs[1..-1].join('.')
         else
-          (at_css('h1') || at_css('h2')).content
+          node = (at_css('h1') || at_css('h2'))
+          return node.content unless node.nil?
+          subpath[/\/([a-z0-9_-]+)\./][1..-2].titleize.sub('Faq', 'FAQ')
         end
       end

lib/docs/filters/mdn/clean_html.rb+90−0

@@ -14,6 +14,8 @@ module Docs · 13 unmodified lines
14	'#Quick_Links',	14	'#Quick_Links',
15	'hr']	15	'hr']
16		16
		17	BROWSER_UNNECESSARY_CLASS_REGEX = /\s*bc-browser[\w_-]+/
		18
17	def call	19	def call
18	css(*REMOVE_NODES).remove	20	css(*REMOVE_NODES).remove
19		21
@@ -61,6 +63,94 @@ module Docs · 41 unmodified lines
61	node.previous_element << node	63	node.previous_element << node
62	end	64	end
63		65
		66	# New compatibility tables
		67
		68	css('.bc-data #Legend + dl', '.bc-data #Legend', '.bc-data #Legend_2 + dl', '.bc-data #Legend_2', '.bc-browser-name').remove
		69
		70	css('abbr.only-icon[title="Full support"]',
		71	'abbr.only-icon[title="Partial support"]',
		72	'abbr.only-icon[title="No support"]',
		73	'abbr.only-icon[title="See implementation notes"]').remove
		74
		75	css('.bc-data .ic-altname', '.bc-data .ic-deprecated', '.bc-data .ic-non-standard', '.bc-data .ic-experimental').each do \|node\|
		76	node.parent.remove
		77	end
		78
		79	css('abbr.only-icon').each do \|node\|
		80	node.replace(node.content)
		81	end
		82
		83	css('.bc-table .bc-platforms td', '.bc-table .bc-browsers td').each do \|node\|
		84	node.name = 'th'
		85	end
		86
		87	css('.bc-data').each do \|node\|
		88	link = node.at_css('.bc-github-link')
		89	prev = node.previous_element
		90	prev = prev.previous_element until prev.name == 'h2'
		91	prev.add_child(link)
		92
		93	node.before(node.children).remove
		94	end
		95
		96	css('.bc-table').each do \|node\|
		97	desktop_table = node
		98
		99	mobile_table = node.dup
		100	desktop_table.after(mobile_table)
		101
		102	if desktop_table.at_css('.bc-platform-server')
		103	server_table = node.dup
		104	mobile_table.after(server_table)
		105	end
		106
		107	desktop_columns = desktop_table.at_css('th.bc-platform-desktop')['colspan'].to_i
		108	mobile_columns = desktop_table.at_css('th.bc-platform-mobile')['colspan'].to_i
		109
		110	desktop_table.css('.bc-platform-mobile').remove
		111	desktop_table.css('.bc-platform-server').remove
		112	desktop_table.css('.bc-browsers th').to_a[(desktop_columns + 1)..-1].each(&:remove)
		113	desktop_table.css('tr:not(.bc-platforms):not(.bc-browsers)').each do \|line\|
		114	line.css('td').to_a[(desktop_columns)..-1].each(&:remove)
		115	end
		116
		117	mobile_table.css('.bc-platform-desktop').remove
		118	mobile_table.css('.bc-platform-server').remove
		119	mobile_table.css('.bc-browsers th').to_a[1..(desktop_columns)].each(&:remove)
		120	mobile_table.css('.bc-browsers th').to_a[(mobile_columns + 1)..-1].each(&:remove)
		121	mobile_table.css('tr:not(.bc-platforms):not(.bc-browsers)').each do \|line\|
		122	line.css('td').to_a[0..(desktop_columns - 1)].each(&:remove)
		123	line.css('td').to_a[(mobile_columns)..-1].each(&:remove)
		124	end
		125
		126	if server_table
		127	server_table.css('.bc-platform-desktop').remove
		128	server_table.css('.bc-platform-mobile').remove
		129	server_table.css('.bc-browsers th').to_a[1..(desktop_columns + mobile_columns)].each(&:remove)
		130	server_table.css('tr:not(.bc-platforms):not(.bc-browsers)').each do \|line\|
		131	line.css('td').to_a[0..(desktop_columns + mobile_columns - 1)].each(&:remove)
		132	end
		133	end
		134	end
		135
		136	# Reduce page size to make the offline bundle smaller.
		137	css('.bc-supports-unknown').remove_attr('class')
		138	css('td[class="bc-platform"], th[class="bc-platform"]').remove_attr('class')
		139	css('td[class="bc-browser"], th[class="bc-browser"]').each do \|node\|
		140	class_name = node['class']
		141	class_name.remove!(BROWSER_UNNECESSARY_CLASS_REGEX)
		142
		143	if class_name.present?
		144	node['class'] = class_name
		145	else
		146	node.remove_attribute('class')
		147	end
		148	end
		149
		150	css('abbr[title*="Compatibility unknown"]').each do \|node\|
		151	node.before(node.children).remove
		152	end
		153
64	doc	154	doc
65	end	155	end
66	end	156	end

lib/docs/filters/pandas/entries.rb+5−3

   class Pandas
     class EntriesFilter < Docs::EntriesFilter
       def get_name
-        if subpath.start_with?('generated')
+        if subpath.start_with?('generated') || (subpath.include?('reference') && !subpath.include?('reference/index'))
-          name = at_css('dt').content.strip
+          name_node = at_css('dt')
+          name_node = at_css('h1') if name_node.nil?
+          name = name_node.content.strip
           name.sub! %r{\(.*}, '()'
           name.remove! %r{\s=.*}
           name.remove! %r{\A(class(method)?) (pandas\.)?}
       end
       def get_type
-        if subpath.start_with?('generated')
+        if subpath.start_with?('generated') || (subpath.include?('reference') && !subpath.include?('reference/index'))
           css('.toctree-l2.current > a').last.content.remove(/\s\(.+?\)/)
         else
           'Manual'

lib/docs/filters/pygame/clean_html.rb+1−1

@@ -85,7 +85,7 @@ module Docs · 84 unmodified lines
85	if initial_header.start_with?(@section)	85	if initial_header.start_with?(@section)
86	sig.content = @section + '.' + sig.text	86	sig.content = @section + '.' + sig.text
87	end	87	end
88	# seperate the signatures on different lines.	88	# separate the signatures on different lines.
89	header.add_child "<br>"	89	header.add_child "<br>"
90	end	90	end
91	end	91	end

lib/docs/filters/python/entries_v2.rb+1−7

         clean_id_attributes
         entries = []
-        css('.class > dt[id]', '.exception > dt[id]', '.attribute > dt[id]').each do |node|
+        css('.class > dt[id]', '.exception > dt[id]', '.attribute > dt[id]', '.data > dt[id]').each do |node|
           entries << [node['id'], node['id']]
         end
-        css('.data > dt[id]').each do |node|
-          if node['id'].split('.').last.upcase! # skip constants
-            entries << [node['id'], node['id']]
-          end
-        end
         css('.function > dt[id]', '.method > dt[id]', '.staticmethod > dt[id]', '.classmethod > dt[id]').each do |node|
           entries << [node['id'] + '()', node['id']]
         end

lib/docs/filters/python/entries_v3.rb+3−7

       def get_type
         return 'Logging' if slug.start_with? 'library/logging'
+        return 'Asynchronous I/O' if slug.start_with? 'library/asyncio'
         type = at_css('.related a[accesskey="U"]').content
       end
       def include_default_entry?
+        return true if slug == 'library/asyncio'
         !at_css('.body > .section:only-child > .toctree-wrapper:last-child') && !type.in?(%w(Superseded))
       end
         clean_id_attributes
         entries = []
-        css('.class > dt[id]', '.exception > dt[id]', '.attribute > dt[id]').each do |node|
+        css('.class > dt[id]', '.exception > dt[id]', '.attribute > dt[id]', '.data > dt[id]').each do |node|
           entries << [node['id'], node['id']]
         end
-        css('.data > dt[id]').each do |node|
-          if node['id'].split('.').last.upcase! # skip constants
-            entries << [node['id'], node['id']]
-          end
-        end
         css('.function > dt[id]', '.method > dt[id]', '.staticmethod > dt[id]', '.classmethod > dt[id]').each do |node|
           entries << [node['id'] + '()', node['id']]
         end

lib/docs/filters/qt/clean_html.rb+1−0

@@ -8,6 +8,7 @@ module Docs · 7 unmodified lines
8	# QML property/method header	8	# QML property/method header
9	css('.qmlproto').each do \|node\|	9	css('.qmlproto').each do \|node\|
10	id = node.at_css('tr')['id']	10	id = node.at_css('tr')['id']
		11	id = node.at_css('a')['name'] if id.blank?
11	node.inner_html = node.at_css('td').inner_html	12	node.inner_html = node.at_css('td').inner_html
12	node.name = 'h3'	13	node.name = 'h3'
13	node['id'] = id	14	node['id'] = id

lib/docs/filters/qt/entries.rb+1−0

@@ -111,6 +111,7 @@ module Docs · 110 unmodified lines
111	css('.qmlproto').each do \|node\|	111	css('.qmlproto').each do \|node\|
112	title = node.content.strip	112	title = node.content.strip
113	id = node.at_css('tr')['id']	113	id = node.at_css('tr')['id']
		114	id = node.at_css('a')['name'] if id.blank?
114		115
115	# Remove options	116	# Remove options
116	title.remove!(%r{^\[.*\] })	117	title.remove!(%r{^\[.*\] })

lib/docs/filters/ramda/clean_html.rb+1−1

@@ -11,7 +11,7 @@ module Docs · 10 unmodified lines
11	css('.card').each do \|card\|	11	css('.card').each do \|card\|
12	title = card.at_css('h2')	12	title = card.at_css('h2')
13	added_in = card.at_css('small')	13	added_in = card.at_css('small')
14	added_in.parent = title	14	added_in.parent = title if added_in.present?
15	end	15	end
16		16
17	css('.params').each do \|node\|	17	css('.params').each do \|node\|

lib/docs/filters/rdoc/clean_html.rb+4−2

         end
         # Add class to differentiate Ruby code from C code
-        css('.method-source-code > pre').each do |node|
+        css('.method-source-code').each do |node|
-          node['class'] = node.at_css('.ruby-keyword') ? 'ruby' : 'c'
+          node.parent.prepend_child(node)
+          pre = node.at_css('pre')
+          pre['class'] = pre.at_css('.ruby-keyword') ? 'ruby' : 'c'
         end
         # Remove code highlighting

lib/docs/filters/rust/clean_html.rb+4−2

+# frozen_string_literal: true
 module Docs
   class Rust
     class CleanHtmlFilter < Filter
           node.remove
         end
-        css('h2 .important-traits', 'h3 .important-traits', 'h4 .important-traits').each do |node|
+        css('.important-traits').to_a.each_with_index do |node, index|
           content = node.at_css('.content.hidden .content')
           node.at_css('.content.hidden').replace(content) if content
-          node.parent.after(node)
+          node.parent.after(node) if node.parent.name.in?(%(h2 h3 h4))
         end
         css('code.content').each do |node|

lib/docs/filters/rust/entries.rb+5−6

           end
         else
           css('.method')
-            .select {|node| !node.at_css('.fnname').nil?}
+            .each_with_object({}) { |node, entries|
-            .map {|node|
+              name = node.at_css('.fnname').try(:content)
-              name = node.at_css('.fnname').content
+              next unless name
               name.prepend "#{self.name}::"
-              [name, node['id']]
+              entries[name] ||= [name, node['id']]
-            }
+            }.values
-            .uniq {|item| item[0]}
         end
       end
     end

lib/docs/filters/salt_stack/clean_html.rb+27−0

@@ -0,0 +1,27 @@
	1	module Docs
	2	class SaltStack
	3	class CleanHtmlFilter < Filter
	4	def call
	5	if root_page?
	6	doc.inner_html = '<h1>SaltStack</h1>'
	7	return doc
	8	end
	9
	10	css('.headerlink').remove
	11
	12	css('div[class^="highlight-"]').each do \|node\|
	13	node.name = 'pre'
	14	node['data-language'] = node['class'].scan(/highlight-([a-z]+)/i)[0][0]
	15	node.content = node.content.strip
	16	end
	17
	18	css('.function > dt').each do \|node\|
	19	node.name = 'h3'
	20	node.content = node.content
	21	end
	22
	23	doc
	24	end
	25	end
	26	end
	27	end

lib/docs/filters/salt_stack/entries.rb+38−0

@@ -0,0 +1,38 @@
	1	module Docs
	2	class SaltStack
	3	class EntriesFilter < Docs::EntriesFilter
	4	SALT_REF_RGX = /salt\.([^\.]+)\.([^\s]+)/
	5
	6	def get_name
	7	header = at_css('h1').content
	8
	9	ref_match = SALT_REF_RGX.match(header)
	10	if ref_match
	11	ns, mod = ref_match.captures
	12	"#{ns}.#{mod}"
	13	else
	14	header
	15	end
	16	end
	17
	18	def get_type
	19	slug.split('/', 3)[1]
	20	end
	21
	22	def include_default_entry?
	23	slug.split('/').last.start_with? 'salt'
	24	end
	25
	26	def additional_entries
	27	entries = []
	28
	29	css('.function > h3').each do \|node\|
	30	name = node.content.remove('salt.').split('(')[0] + '()'
	31	entries << [name, node['id']]
	32	end
	33
	34	entries
	35	end
	36	end
	37	end
	38	end

lib/docs/filters/scala/clean_html.rb+109−0

@@ -0,0 +1,109 @@
	1	module Docs
	2	class Scala
	3	class CleanHtmlFilter < Filter
	4	def call
	5	@doc = at_css('#content')
	6
	7	always
	8	add_title
	9
	10	doc
	11	end
	12
	13	def always
	14	# Remove deprecated sections
	15	css('.members').each do \|members\|
	16	header = members.at_css('h3')
	17	members.remove if header.text.downcase.include? 'deprecate'
	18	end
	19
	20	css('#mbrsel, #footer').remove
	21
	22	css('.diagram-container').remove
	23	css('.toggleContainer > .toggle').each do \|node\|
	24	title = node.at_css('span')
	25	next if title.nil?
	26
	27	content = node.at_css('.hiddenContent')
	28	next if content.nil?
	29
	30	title.name = 'dt'
	31
	32	content.remove_attribute('class')
	33	content.remove_attribute('style')
	34	content.name = 'dd'
	35
	36	attributes = at_css('.attributes')
	37	unless attributes.nil?
	38	title.parent = attributes
	39	content.parent = attributes
	40	end
	41	end
	42
	43	signature = at_css('#signature')
	44	signature.replace "<h2 id=\"signature\">#{signature.inner_html}</h2>"
	45
	46	css('div.members > h3').each do \|node\|
	47	node.name = 'h2'
	48	end
	49
	50	css('div.members > ol').each do \|list\|
	51	list.css('li').each do \|li\|
	52	h3 = doc.document.create_element 'h3'
	53	h3['id'] = li['name'].rpartition('#').last unless li['name'].nil?
	54
	55	li.prepend_child h3
	56	li.css('.shortcomment').remove
	57
	58	modifier = li.at_css('.modifier_kind')
	59	modifier.parent = h3 unless modifier.nil?
	60
	61	kind = li.at_css('.modifier_kind .kind')
	62	kind.content = kind.content + ' ' unless kind.nil?
	63
	64	symbol = li.at_css('.symbol')
	65	symbol.parent = h3 unless symbol.nil?
	66
	67	li.swap li.children
	68	end
	69
	70	list.swap list.children
	71	end
	72
	73	css('.fullcomment pre, .fullcommenttop pre').each do \|pre\|
	74	pre['data-language'] = 'scala'
	75	pre.content = pre.content
	76	end
	77
	78	# Sections of the documentation which do not seem useful
	79	%w(#inheritedMembers #groupedMembers .permalink .hiddenContent .material-icons).each do \|selector\|
	80	css(selector).remove
	81	end
	82
	83	# Things that are not shown on the site, like deprecated members
	84	css('li[visbl=prt]').remove
	85	end
	86
	87	def add_title
	88	css('.permalink').remove
	89
	90	definition = at_css('#definition')
	91	return if definition.nil?
	92
	93	type_full_name = {a: 'Annotation', c: 'Class', t: 'Trait', o: 'Object', p: 'Package'}
	94	type = type_full_name[definition.at_css('.big-circle').text.to_sym]
	95	name = CGI.escapeHTML definition.at_css('h1').text
	96
	97	package = definition.at_css('#owner').text rescue ''
	98	package = package + '.' unless name.empty? \|\| package.empty?
	99
	100	other = definition.at_css('.morelinks').dup
	101	other_content = other ? "<h3>#{other.to_html}</h3>" : ''
	102
	103	title_content = root_page? ? 'Package root' : "#{type} #{package}#{name}".strip
	104	title = "<h1>#{title_content}</h1>"
	105	definition.replace title + other_content
	106	end
	107	end
	108	end
	109	end

lib/docs/filters/scala/entries.rb+103−0

@@ -0,0 +1,103 @@
	1	module Docs
	2	class Scala
	3	class EntriesFilter < Docs::EntriesFilter
	4	REPLACEMENTS = {
	5	'$eq' => '=',
	6	'$colon' => ':',
	7	'$less' => '<',
	8	}
	9
	10	def get_name
	11	if is_package?
	12	symbol = at_css('#definition h1')
	13	symbol ? symbol.text.gsub(/\W+/, '') : "package"
	14	else
	15	name = slug.split('/').last
	16
	17	# Some objects have inner objects, show ParentObject$.ChildObject$ instead of ParentObject$$ChildObject$
	18	name = name.gsub('$$', '$.')
	19
	20	REPLACEMENTS.each do \|key, value\|
	21	name = name.gsub(key, value)
	22	end
	23
	24	# If a dollar sign is used as separator between two characters, replace it with a dot
	25	name.gsub(/([^$.])\$([^$.])/, '\1.\2')
	26	end
	27	end
	28
	29	def get_type
	30	# if this entry is for a package, we group the package under the parent package
	31	if is_package?
	32	parent_package
	33	# otherwise, group it under the regular package name
	34	else
	35	package_name
	36	end
	37	end
	38
	39	def include_default_entry?
	40	true
	41	end
	42
	43	def additional_entries
	44	entries = []
	45
	46	full_name = "#{type}.#{name}".remove('$')
	47	css(".members li[name^=\"#{full_name}\"]").each do \|node\|
	48	# Ignore packages
	49	kind = node.at_css('.modifier_kind > .kind')
	50	next if !kind.nil? && kind.content == 'package'
	51
	52	# Ignore deprecated members
	53	next unless node.at_css('.symbol > .name.deprecated').nil?
	54
	55	id = node['name'].rpartition('#').last
	56	member_name = node.at_css('.name')
	57
	58	# Ignore members only existing of hashtags, we can't link to that
	59	next if member_name.nil? \|\| member_name.content.strip.remove('#').blank?
	60
	61	member = "#{name}.#{member_name.content}()"
	62	entries << [member, id]
	63	end
	64
	65	entries
	66	end
	67
	68	private
	69
	70	# For the package name, we use the slug rather than parsing the package
	71	# name from the HTML because companion object classes may be broken out into
	72	# their own entries (by the source documentation). When that happens,
	73	# we want to group these classes (like `scala.reflect.api.Annotations.Annotation`)
	74	# under the package name, and not the fully-qualfied name which would
	75	# include the companion object.
	76	def package_name
	77	name = package_drop_last(slug_parts)
	78	name.empty? ? '_root_' : name
	79	end
	80
	81	def parent_package
	82	parent = package_drop_last(package_name.split('.'))
	83	parent.empty? ? '_root_' : parent
	84	end
	85
	86	def package_drop_last(parts)
	87	parts[0...-1].join('.')
	88	end
	89
	90	def slug_parts
	91	slug.split('/')
	92	end
	93
	94	def owner
	95	at_css('#owner')
	96	end
	97
	98	def is_package?
	99	slug.ends_with?('index') \|\| slug.ends_with?('package')
	100	end
	101	end
	102	end
	103	end

lib/docs/filters/vue_router/clean_html.rb+21−0

@@ -0,0 +1,21 @@
	1	module Docs
	2	class VueRouter
	3	class CleanHtmlFilter < Filter
	4	def call
	5	@doc = at_css('.content')
	6
	7	# Remove unneeded elements
	8	css('.bit-sponsor, .header-anchor').remove
	9
	10	css('.custom-block').each do \|node\|
	11	node.name = 'blockquote'
	12
	13	title = node.at_css('.custom-block-title')
	14	title.name = 'strong' unless title.nil?
	15	end
	16
	17	doc
	18	end
	19	end
	20	end
	21	end

lib/docs/filters/vue_router/entries.rb+71−0

@@ -0,0 +1,71 @@
	1	module Docs
	2	class VueRouter
	3	class EntriesFilter < Docs::EntriesFilter
	4	def get_name
	5	name = at_css('h1').content
	6	name.remove! '# '
	7	name
	8	end
	9
	10	def get_type
	11	return 'Other Guides' if subpath.start_with?('guide/advanced')
	12	return 'Basic Guides' if subpath.start_with?('guide') \|\| subpath.start_with?('installation')
	13	'API Reference'
	14	end
	15
	16	def include_default_entry?
	17	name != 'API Reference'
	18	end
	19
	20	def additional_entries
	21	return [] unless subpath.start_with?('api')
	22
	23	entries = [
	24	['<router-link>', 'router-link', 'API Reference'],
	25	['<router-view>', 'router-view', 'API Reference'],
	26	['$route', 'the-route-object', 'API Reference'],
	27	['Component Injections', 'component-injections', 'API Reference']
	28	]
	29
	30	css('h3').each do \|node\|
	31	entry_name = node.content.strip
	32
	33	# Get the previous h2 title
	34	title = node
	35	title = title.previous_element until title.name == 'h2'
	36	title = title.content.strip
	37	title.remove! '# '
	38
	39	entry_name.remove! '# '
	40
	41	case title
	42	when 'Router Construction Options'
	43	entry_name = "RouterOptions.#{entry_name}"
	44	when '<router-view> Props'
	45	entry_name = "<router-view> `#{entry_name}` prop"
	46	when '<router-link> Props'
	47	entry_name = "<router-link> `#{entry_name}` prop"
	48	when 'Router Instance Methods'
	49	entry_name = "#{entry_name}()"
	50	end
	51
	52	entry_name = entry_name.split(' API ')[0] if entry_name.start_with?('v-slot')
	53
	54	unless title == "Component Injections" \|\| node['id'] == 'route-object-properties'
	55	entries << [entry_name, node['id'], 'API Reference']
	56	end
	57	end
	58
	59	css('#route-object-properties + ul > li > p:first-child > strong').each do \|node\|
	60	entry_name = node.content.strip
	61	id = "route-object-#{entry_name.remove('$route.')}"
	62
	63	node['id'] = id
	64	entries << [entry_name, node['id'], 'API Reference']
	65	end
	66
	67	entries
	68	end
	69	end
	70	end
	71	end

lib/docs/filters/wordpress/clean_html.rb+39−0

@@ -0,0 +1,39 @@
	1	module Docs
	2	class Wordpress
	3	class CleanHtmlFilter < Filter
	4	def call
	5	if root_page?
	6	doc.inner_html = '<h1>WordPress</h1>'
	7	return doc
	8	end
	9
	10	article = at_css('article[id^="post-"]')
	11	@doc = at_css('article[id^="post-"]') unless article.nil?
	12
	13	css('hr', '.screen-reader-text', '.table-of-contents',
	14	'.anchor', '.toc-jump', '.source-code-links', '.user-notes',
	15	'.show-more', '.hide-more').remove
	16
	17	br = /<br\s?\/?>/i
	18
	19	header = at_css('h1')
	20	header.content = header.content.strip
	21	doc.prepend_child header
	22
	23	# Add PHP code highlighting
	24	css('pre').each do \|node\|
	25	node['data-language'] = 'php'
	26	end
	27
	28	css('.source-code-container').each do \|node\|
	29	node.name = 'pre'
	30	node.inner_html = node.inner_html.gsub(br, "\n")
	31	node.content = node.content.strip
	32	node['data-language'] = 'php'
	33	end
	34
	35	doc
	36	end
	37	end
	38	end
	39	end

lib/docs/filters/wordpress/entries.rb+19−0

@@ -0,0 +1,19 @@
	1	module Docs
	2	class Wordpress
	3	class EntriesFilter < Docs::EntriesFilter
	4	def get_name
	5	at_css('.breadcrumbs .trail-end').content
	6	end
	7
	8	def get_type
	9	if subpath.starts_with?('classes')
	10	'Classes'
	11	elsif subpath.starts_with?('hooks')
	12	'Hooks'
	13	elsif subpath.starts_with?('functions')
	14	'Functions'
	15	end
	16	end
	17	end
	18	end
	19	end

lib/docs/scrapers/angular.rb+4−0

@@ -155,6 +155,10 @@ module Docs · 154 unmodified lines
155	end	155	end
156	end	156	end
157		157
		158	def get_latest_version(opts)
		159	get_npm_version('@angular/core', opts)
		160	end
		161
158	private	162	private
159		163
160	def parse(response)	164	def parse(response)

lib/docs/scrapers/angularjs.rb+16−3

     end
     options[:only_patterns] = [%r{\Aapi}, %r{\Aguide}]
-    options[:skip] = %w(api/ng.html)
+    options[:skip] = %w(api/ng.html guide/tutorial/step_14.html guide/api.html guide/tutorial/.html)
+    options[:skip_patterns] = [
+      /error\/\$compile/,
+      /misc/,
+    ]
     options[:attribution] = <<-HTML
-      &copy; 2010&ndash;2017 Google, Inc.<br>
+      &copy; 2010&ndash;2018 Google, Inc.<br>
       Licensed under the Creative Commons Attribution License 4.0.
     HTML
       capybara.execute_script("return document.querySelector('.side-navigation').innerHTML")
     end
+    version '1.7' do
+      self.release = '1.7.8'
+      self.base_url = "https://code.angularjs.org/#{release}/docs/partials/"
+    end
     version '1.6' do
-      self.release = '1.6.6'
+      self.release = '1.6.9'
       self.base_url = "https://code.angularjs.org/#{release}/docs/partials/"
     end
       self.release = '1.2.32'
       self.base_url = "https://code.angularjs.org/#{release}/docs/partials/"
     end
+    def get_latest_version(opts)
+      get_npm_version('angular', opts)
+    end
   end
 end

lib/docs/scrapers/ansible.rb+31−50

@@ -11,72 +11,47 @@ module Docs · 10 unmodified lines
11		11
12	options[:attribution] = <<-HTML	12	options[:attribution] = <<-HTML
13	© 2012–2018 Michael DeHaan<br>	13	© 2012–2018 Michael DeHaan<br>
14	© 2018 Red Hat, Inc.<br>	14	© 2018–2019 Red Hat, Inc.<br>
15	Licensed under the GNU General Public License version 3.	15	Licensed under the GNU General Public License version 3.
16	HTML	16	HTML
17		17
18	version '2.7' do	18	options[:skip] = %w(
19	self.release = '2.7.1'	19	installation_guide/index.html
20	self.base_url = 'https://docs.ansible.com/ansible/2.7/'	20	reference_appendices/glossary.html
		21	reference_appendices/faq.html
		22	reference_appendices/tower.html
		23	user_guide/quickstart.html
		24	modules/modules_by_category.html
		25	modules/list_of_all_modules.html)
21		26
22	options[:skip] = %w(	27	options[:skip_patterns] = [
23	installation_guide/index.html	28	/\Acommunity.*/i,
24	reference_appendices/glossary.html	29	/\Adev_guide.*/i,
25	reference_appendices/faq.html	30	/\Aroadmap.*/i,
26	reference_appendices/tower.html	31	]
27	user_guide/quickstart.html	32
28	modules/modules_by_category.html	33	version '2.8' do
29	modules/list_of_all_modules.html)	34	self.release = '2.8.3'
		35	self.base_url = 'https://docs.ansible.com/ansible/2.7/'
		36	end
30		37
31	options[:skip_patterns] = [	38	version '2.7' do
32	/\Acommunity.*/i,	39	self.release = '2.7.12'
33	/\Adev_guide.*/i,	40	self.base_url = 'https://docs.ansible.com/ansible/2.7/'
34	/\Aroadmap.*/i,
35	]
36	end	41	end
37		42
38	version '2.6' do	43	version '2.6' do
39	self.release = '2.6.7'	44	self.release = '2.6.18'
40	self.base_url = 'https://docs.ansible.com/ansible/2.6/'	45	self.base_url = 'https://docs.ansible.com/ansible/2.6/'
41
42	options[:skip] = %w(
43	installation_guide/index.html
44	reference_appendices/glossary.html
45	reference_appendices/faq.html
46	reference_appendices/tower.html
47	user_guide/quickstart.html
48	modules/modules_by_category.html
49	modules/list_of_all_modules.html)
50
51	options[:skip_patterns] = [
52	/\Acommunity.*/i,
53	/\Adev_guide.*/i,
54	/\Aroadmap.*/i,
55	]
56	end	46	end
57		47
58	version '2.5' do	48	version '2.5' do
59	self.release = '2.5.3'	49	self.release = '2.5.15'
60	self.base_url = 'https://docs.ansible.com/ansible/2.5/'	50	self.base_url = 'https://docs.ansible.com/ansible/2.5/'
61
62	options[:skip] = %w(
63	installation_guide/index.html
64	reference_appendices/glossary.html
65	reference_appendices/faq.html
66	reference_appendices/tower.html
67	user_guide/quickstart.html
68	modules/modules_by_category.html
69	modules/list_of_all_modules.html)
70
71	options[:skip_patterns] = [
72	/\Acommunity.*/i,
73	/\Adev_guide.*/i,
74	/\Aroadmap.*/i,
75	]
76	end	51	end
77		52
78	version '2.4' do	53	version '2.4' do
79	self.release = '2.4.3'	54	self.release = '2.4.6'
80	self.base_url = 'https://docs.ansible.com/ansible/2.4/'	55	self.base_url = 'https://docs.ansible.com/ansible/2.4/'
81		56
82	options[:skip] = %w(	57	options[:skip] = %w(
@@ -86,6 +61,12 @@ module Docs · 3 unmodified lines
86	tower.html	61	tower.html
87	quickstart.html	62	quickstart.html
88	list_of_all_modules.html)	63	list_of_all_modules.html)
		64	options[:skip_patterns] = []
		65	end
		66
		67	def get_latest_version(opts)
		68	doc = fetch_doc('https://docs.ansible.com/ansible/latest/index.html', opts)
		69	doc.at_css('.DocSiteProduct-CurrentVersion').content.strip
89	end	70	end
90	end	71	end
91	end	72	end

lib/docs/scrapers/apache.rb+5−0

       &copy; 2018 The Apache Software Foundation<br>
       Licensed under the Apache License, Version 2.0.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('http://httpd.apache.org/docs/', opts)
+      doc.at_css('#apcontents > ul a')['href'][0...-1]
+    end
   end
 end

lib/docs/scrapers/apache_pig.rb+5−0

       self.base_url = "https://pig.apache.org/docs/r#{release}/"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://pig.apache.org/', opts)
+      item = doc.at_css('div[id="menu_1.2"] > .menuitem:last-child')
+      item.content.strip.sub(/Release /, '')
+    end
   end
 end

lib/docs/scrapers/async.rb+5−0

       &copy; 2010&ndash;2018 Caolan McMahon<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://caolan.github.io/async/v3/', opts)
+      doc.at_css('#version-dropdown > a').content.strip[1..-1]
+    end
   end
 end

lib/docs/scrapers/babel.rb+5−0

     stub '' do
       '<div></div>'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://babeljs.io/docs/en/', opts)
+      doc.at_css('a[href="/versions"] > h3').content
+    end
   end
 end

lib/docs/scrapers/backbone.rb+5−0

       &copy; 2010&ndash;2016 Jeremy Ashkenas, DocumentCloud<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://backbonejs.org/', opts)
+      doc.at_css('.version').content[1...-1]
+    end
   end
 end

lib/docs/scrapers/bash.rb+6−1

 module Docs
   class Bash < UrlScraper
     self.type = 'bash'
-    self.release = '4.4'
+    self.release = '5.0'
     self.base_url = 'https://www.gnu.org/software/bash/manual'
     self.root_path = '/html_node/index.html'
     self.links = {
       Copyright &copy; 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.<br>
       Licensed under the GNU Free Documentation License.
     HTML
+    def get_latest_version(opts)
+      body = fetch('https://www.gnu.org/software/bash/manual/html_node/index.html', opts)
+      body.scan(/, Version ([0-9.]+)/)[0][0][0...-1]
+    end
   end
 end

lib/docs/scrapers/bluebird.rb+4−0

       &copy; 2013&ndash;2017 Petka Antonov<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('bluebird', opts)
+    end
   end
 end

lib/docs/scrapers/bootstrap.rb+10−4

     options[:trailing_slash] = true
+    # https://github.com/twbs/bootstrap/blob/master/LICENSE
     options[:attribution] = <<-HTML
-      &copy; 2011&ndash;2018 Twitter, Inc.<br>
+      &copy; 2011&ndash;2019 Twitter, Inc.<br>
-      &copy; 2011&ndash;2018 The Bootstrap Authors<br>
+      &copy; 2011&ndash;2019 The Bootstrap Authors<br>
       Code licensed under the MIT License.<br>
       Documentation licensed under the Creative Commons Attribution License v3.0.
     HTML
     version '4' do
-      self.release = '4.1.3'
+      self.release = '4.3.1'
-      self.base_url = 'https://getbootstrap.com/docs/4.1/'
+      self.base_url = 'https://getbootstrap.com/docs/4.3/'
       self.root_path = 'getting-started/introduction/'
       html_filters.push 'bootstrap/entries_v4', 'bootstrap/clean_html_v4'
       options[:only] = %w(getting-started/ css/ components/ javascript/)
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://getbootstrap.com/', opts)
+      doc.at_css('#bd-versions').content.strip[1..-1]
+    end
   end
 end

lib/docs/scrapers/bottle.rb+6−0

@@ -27,5 +27,11 @@ module Docs · 26 unmodified lines
27	self.release = '0.11.7'	27	self.release = '0.11.7'
28	self.base_url = "https://bottlepy.org/docs/#{self.version}/"	28	self.base_url = "https://bottlepy.org/docs/#{self.version}/"
29	end	29	end
		30
		31	def get_latest_version(opts)
		32	doc = fetch_doc('https://bottlepy.org/docs/stable/', opts)
		33	label = doc.at_css('.sphinxsidebarwrapper > ul > li > b')
		34	label.content.sub(/Bottle /, '')
		35	end
30	end	36	end
31	end	37	end

lib/docs/scrapers/bower.rb+4−0

       &copy; 2018 Bower contributors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('bower', opts)
+    end
   end
 end

lib/docs/scrapers/c.rb+7−1

 module Docs
   class C < FileScraper
     self.type = 'c'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/c'
     self.base_url = 'http://en.cppreference.com/w/c/'
     self.root_path = 'header.html'
       Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts)
+      link = doc.at_css('a[title^="File:"]')
+      date = link.content.scan(/(\d+)\./)[0][0]
+      DateTime.strptime(date, '%Y%m%d').to_time.to_i
+    end
     private
     def file_path_for(*)

lib/docs/scrapers/cakephp.rb+5−0

       self.base_url = 'https://api.cakephp.org/2.7/'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://api.cakephp.org/3.7/', opts)
+      doc.at_css('.version-picker .dropdown-toggle').content.strip
+    end
     private
     def parse(response)

lib/docs/scrapers/chai.rb+4−0

       &copy; 2016 Chai.js Assertion Library<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('chai', opts)
+    end
   end
 end

lib/docs/scrapers/chef.rb+5−0

       options[:only_patterns] = [/\A#{client_path}\//, /\A#{server_path}\//]
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://downloads.chef.io/chef', opts)
+      doc.at_css('h1.product-heading > span').content.strip
+    end
   end
 end

lib/docs/scrapers/clojure.rb+14−4

       Licensed under the Eclipse Public License 1.0.
     HTML
-    version '1.9' do
+    version '1.10' do
-      self.release = '1.9'
+      self.release = '1.10 (stable)'
       self.base_url = 'https://clojure.github.io/clojure/'
     end
+    version '1.9' do
+      self.release = '1.9 (legacy)'
+      self.base_url = 'https://clojure.github.io/clojure/branch-clojure-1.9.0/'
+    end
     version '1.8' do
-      self.release = '1.8'
+      self.release = '1.8 (legacy)'
       self.base_url = 'https://clojure.github.io/clojure/branch-clojure-1.8.0/'
     end
     version '1.7' do
-      self.release = '1.7'
+      self.release = '1.7 (legacy)'
       self.base_url = 'https://clojure.github.io/clojure/branch-clojure-1.7.0/'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('http://clojure.github.io/clojure/index.html', opts)
+      doc.at_css('#header-version').content[1..-1]
+    end
   end
 end

lib/docs/scrapers/cmake.rb+32−11

     options[:title] = false
     options[:root_title] = 'CMake Reference Documentation'
     options[:skip] = %w(release/index.html genindex.html search.html)
-    options[:skip_patterns] = [/\Agenerator/, /\Ainclude/, /\Arelease/]
+    options[:skip_patterns] = [/\Agenerator/, /\Acpack_gen/, /\Ainclude/, /\Arelease/]
     options[:attribution] = <<-HTML
-      &copy; 2000&ndash;2018 Kitware, Inc. and Contributors<br>
+      &copy; 2000&ndash;2019 Kitware, Inc. and Contributors<br>
       Licensed under the BSD 3-clause License.
     HTML
+    version '3.15' do
+      self.release = '3.15.2'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
+    end
+    version '3.14' do
+      self.release = '3.14.6'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
+    end
+    version '3.13' do
+      self.release = '3.13.5'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
+    end
     version '3.12' do
-      self.release = '3.12.1'
+      self.release = '3.12.4'
-      self.base_url = 'https://cmake.org/cmake/help/v3.12/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
     end
     version '3.11' do
       self.release = '3.11.4'
-      self.base_url = 'https://cmake.org/cmake/help/v3.11/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
     end
     version '3.10' do
       self.release = '3.10.3'
-      self.base_url = 'https://cmake.org/cmake/help/v3.10/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
     end
     version '3.9' do
       self.release = '3.9.6'
-      self.base_url = 'https://cmake.org/cmake/help/v3.9/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
     end
     version '3.8' do
       self.release = '3.8.2'
-      self.base_url = 'https://cmake.org/cmake/help/v3.8/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
     end
     version '3.7' do
       self.release = '3.7.2'
-      self.base_url = 'https://cmake.org/cmake/help/v3.7/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
     end
     version '3.6' do
       self.release = '3.6.3'
-      self.base_url = 'https://cmake.org/cmake/help/v3.6/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
     end
     version '3.5' do
       self.release = '3.5.2'
-      self.base_url = 'https://cmake.org/cmake/help/v3.5/'
+      self.base_url = "https://cmake.org/cmake/help/v#{self.version}/"
+    end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://cmake.org/documentation/', opts)
+      link = doc.at_css('.entry-content ul > li > strong > a > big')
+      link.content.scan(/([0-9.]+)/)[0][0]
     end
   end
 end

lib/docs/scrapers/codeception.rb+5−0

       &copy; 2011 Michael Bodnarchuk and contributors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://codeception.com/changelog', opts)
+      doc.at_css('#page > h4').content
+    end
   end
 end

lib/docs/scrapers/codeceptjs.rb+4−0

       &copy; 2015 DavertMik &lt;davert@codegyre.com&gt; (http://codegyre.com)<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('codeceptjs', opts)
+    end
   end
 end

lib/docs/scrapers/codeigniter.rb+6−0

@@ -38,5 +38,11 @@ module Docs · 37 unmodified lines
38	version '3' do	38	version '3' do
39	self.release = '3.1.8'	39	self.release = '3.1.8'
40	end	40	end
		41
		42	def get_latest_version(opts)
		43	doc = fetch_doc('https://codeigniter.com/user_guide/changelog.html', opts)
		44	header = doc.at_css('#change-log h2')
		45	header.content.scan(/([0-9.]+)/)[0][0]
		46	end
41	end	47	end
42	end	48	end

lib/docs/scrapers/coffeescript.rb+4−0

@@ -30,5 +30,9 @@ module Docs · 29 unmodified lines
30		30
31	options[:container] = '.container'	31	options[:container] = '.container'
32	end	32	end
		33
		34	def get_latest_version(opts)
		35	get_npm_version('coffeescript', opts)
		36	end
33	end	37	end
34	end	38	end

lib/docs/scrapers/composer.rb+28−0

@@ -0,0 +1,28 @@
	1	module Docs
	2	class Composer < UrlScraper
	3	self.type = 'simple'
	4	self.release = '1.9.0'
	5	self.base_url = 'https://getcomposer.org/doc/'
	6	self.links = {
	7	home: 'https://getcomposer.org',
	8	code: 'https://github.com/composer/composer'
	9	}
	10
	11	html_filters.push 'composer/clean_html', 'composer/entries'
	12
	13	options[:container] = '#main'
	14
	15	options[:skip_patterns] = [
	16	/^faqs/
	17	]
	18
	19	options[:attribution] = <<-HTML
	20	© Nils Adermann, Jordi Boggiano<br>
	21	Licensed under the MIT License.
	22	HTML
	23
	24	def get_latest_version(opts)
	25	get_latest_github_release('composer', 'composer', opts)
	26	end
	27	end
	28	end

lib/docs/scrapers/cordova.rb+10−0

@@ -42,5 +42,15 @@ module Docs · 41 unmodified lines
42	self.release = '6.5.0'	42	self.release = '6.5.0'
43	self.base_url = 'https://cordova.apache.org/docs/en/6.x/'	43	self.base_url = 'https://cordova.apache.org/docs/en/6.x/'
44	end	44	end
		45
		46	def get_latest_version(opts)
		47	doc = fetch_doc('https://cordova.apache.org/docs/en/latest/', opts)
		48
		49	label = doc.at_css('#versionDropdown').content.strip
		50	version = label.scan(/([0-9.]+)/)[0][0]
		51	version = version[0...-1] if version.end_with?('.')
		52
		53	version
		54	end
45	end	55	end
46	end	56	end

lib/docs/scrapers/cpp.rb+8−1

     self.name = 'C++'
     self.slug = 'cpp'
     self.type = 'c'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/cpp'
     self.base_url = 'http://en.cppreference.com/w/cpp/'
     self.root_path = 'header.html'
       Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
     HTML
+    # Same as get_latest_version in lib/docs/scrapers/c.rb
+    def get_latest_version(opts)
+      doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts)
+      link = doc.at_css('a[title^="File:"]')
+      date = link.content.scan(/(\d+)\./)[0][0]
+      DateTime.strptime(date, '%Y%m%d').to_time.to_i
+    end
     private
     def file_path_for(*)

lib/docs/scrapers/crystal.rb+5−0

         HTML
       end
     }
+    def get_latest_version(opts)
+      body = fetch('https://crystal-lang.org/api', opts)
+      body.scan(/Crystal Docs ([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/cypress.rb+36−0

@@ -0,0 +1,36 @@
	1	# frozen_string_literal: true
	2
	3	module Docs
	4	class Cypress < UrlScraper
	5	self.name = 'Cypress'
	6	self.type = 'cypress'
	7	self.release = '3.4.1'
	8	self.base_url = 'https://docs.cypress.io'
	9	self.root_path = '/api/api/table-of-contents.html'
	10	self.links = {
	11	home: 'https://www.cypress.io/',
	12	code: 'https://github.com/cypress-io/cypress',
	13	}
	14
	15	html_filters.push 'cypress/entries', 'cypress/clean_html'
	16
	17	options[:container] = '#content'
	18	options[:max_image_size] = 300_000
	19	options[:include_default_entry] = true
	20
	21	options[:skip_patterns] = [/examples\//]
	22	options[:skip_link] = ->(link) {
	23	href = link.attr(:href)
	24	href.nil? ? true : EntriesFilter::SECTIONS.none? { \|section\| href.match?("/#{section}/") }
	25	}
	26
	27	options[:attribution] = <<-HTML
	28	© 2017 Cypress.io<br>
	29	Licensed under the MIT License.
	30	HTML
	31
	32	def get_latest_version(opts)
	33	get_latest_github_release('cypress-io', 'cypress', opts)
	34	end
	35	end
	36	end

lib/docs/scrapers/d.rb+5−0

     def initial_urls
       %w(https://dlang.org/phobos/index.html https://dlang.org/spec/intro.html)
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://dlang.org/changelog/', opts)
+      doc.at_css('#content > ul > li:nth-child(2) > a')['id']
+    end
   end
 end

lib/docs/scrapers/d3.rb+4−0

       options[:root_title] = 'D3.js'
       options[:only_patterns] = [/\.md\z/]
     end
+    def get_latest_version(opts)
+      get_npm_version('d3', opts)
+    end
   end
 end

lib/docs/scrapers/dart.rb+6−2

     version '2' do
       self.release = '2.0.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Dart2'
       self.base_url = "https://api.dartlang.org/stable/#{release}/"
     end
     version '1' do
       self.release = '1.24.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Dart1'
       self.base_url = "https://api.dartlang.org/stable/#{release}/"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://api.dartlang.org/', opts)
+      label = doc.at_css('footer > span').content.strip
+      label.sub(/Dart /, '')
+    end
   end
 end

lib/docs/scrapers/django.rb+13−9

       Licensed under the BSD License.
     HTML
+    version '2.2' do
+      self.release = '2.2.4'
+      self.base_url = 'https://docs.djangoproject.com/en/2.2/'
+    end
     version '2.1' do
-      self.release = '2.1.0'
+      self.release = '2.1.11'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Django21'
       self.base_url = 'https://docs.djangoproject.com/en/2.1/'
     end
     version '2.0' do
-      self.release = '2.0.7'
+      self.release = '2.0.13'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Django20'
       self.base_url = 'https://docs.djangoproject.com/en/2.0/'
     end
     version '1.11' do
-      self.release = '1.11.9'
+      self.release = '1.11.23'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Django111'
       self.base_url = 'https://docs.djangoproject.com/en/1.11/'
     end
     version '1.10' do
       self.release = '1.10.8'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Django110'
       self.base_url = 'https://docs.djangoproject.com/en/1.10/'
     end
     version '1.9' do
       self.release = '1.9.13'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Django19'
       self.base_url = 'https://docs.djangoproject.com/en/1.9/'
     end
     version '1.8' do
       self.release = '1.8.18'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Django18'
       self.base_url = 'https://docs.djangoproject.com/en/1.8/'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.djangoproject.com/', opts)
+      doc.at_css('#doc-versions > li.current > span > strong').content
+    end
   end
 end

lib/docs/scrapers/docker.rb+153−26

@@ -15,15 +15,15 @@ module Docs · 14 unmodified lines
15	end	15	end
16		16
17	options[:attribution] = <<-HTML	17	options[:attribution] = <<-HTML
18	© 2017 Docker, Inc.<br>	18	© 2019 Docker, Inc.<br>
19	Licensed under the Apache License, Version 2.0.<br>	19	Licensed under the Apache License, Version 2.0.<br>
20	Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries.<br>	20	Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries.<br>
21	Docker, Inc. and other parties may also have trademark rights in other terms used herein.	21	Docker, Inc. and other parties may also have trademark rights in other terms used herein.
22	HTML	22	HTML
23		23
24	version '17' do	24	version '19' do
25	self.release = '17.06'	25	self.release = '19.03'
26	self.base_url = 'https://docs.docker.com/'	26	self.base_url = "https://docs.docker.com/"
27		27
28	html_filters.push 'docker/entries', 'docker/clean_html'	28	html_filters.push 'docker/entries', 'docker/clean_html'
29		29
@@ -33,29 +33,150 @@ module Docs · 3 unmodified lines
33	options[:skip_patterns] = [/\Aengine\/api\/v/, /glossary/, /docker-ee/]	33	options[:skip_patterns] = [/\Aengine\/api\/v/, /glossary/, /docker-ee/]
34		34
35	options[:replace_paths] = {	35	options[:replace_paths] = {
36	'engine/installation/linux/docker-ee/linux-postinstall/' => 'engine/installation/linux/linux-postinstall/',	36	'install/linux/ubuntu/' => 'install/linux/docker-ce',
37	'engine/installation/linux/docker-ee/' => 'engine/installation/',	37	'get-started/part1' => 'get-started',
38	'engine/installation/linux/docker-ce/' => 'engine/installation/',	38	'engine/installation/' => 'install',
39	'engine/installation/linux/' => 'engine/installation/',	39	'engine/installation/linux/linux-postinstall/' => 'install/linux',
40	'engine/installation/windows/' => 'engine/installation/',	40	'compose/overview/' => 'compose',
41	'engine/userguide/intro/' => 'engine/userguide/',	41	'docker-cloud/' => 'docker-hub',
42	'engine/tutorials/dockervolumes/' => 'engine/admin/volumes/volumes/',	42	'datacenter/install/linux/' => 'ee',
43	'engine/getstarted/' => 'get-started/',	43	'engine/userguide/' => 'config/daemon',
44	'engine/tutorials/dockerimages/' => 'get-started/',	44	'engine/admin/' => 'config/daemon',
45	'engine/tutorials/dockerrepos/' => 'get-started/',	45	'opensource/get-help/' => 'opensource',
46	'engine/admin/host_integration/' => 'engine/admin/start-containers-automatically/',	46	'engine/tutorials/dockerimages/' => 'get-started',
47	'engine/installation/linux/rhel/' => 'engine/installation/linux/docker-ee/rhel/',	47	'engine/admin/volumes/bind-mounts/' => 'storage',
48	'engine/installation/linux/ubuntulinux/' => 'engine/installation/linux/docker-ee/ubuntu/',	48	'engine/tutorials/dockervolumes/' => 'storage',
49	'engine/installation/linux/suse/' => 'engine/installation/linux/docker-ee/suse/',	49	'engine/admin/volumes/volumes/' => 'storage',
50	'engine/admin/logging/' => 'engine/admin/logging/view_container_logs/',	50	'engine/userguide/labels-custom-metadata/' => 'config',
51	'engine/swarm/how-swarm-mode-works/' => 'engine/swarm/how-swarm-mode-works/nodes/',	51	'engine/userguide/eng-image/multistage-build/' => 'develop/develop-images',
52	'engine/installation/binaries/' => 'engine/installation/linux/docker-ce/binaries/',	52	'engine/swarm/networking/' => 'network',
		53	'engine/admin/resource_constraints/' => 'config/containers',
		54	'engine/admin/logging/overview/' => 'config/containers/logging',
		55	'engine/userguide/eng-image/dockerfile_best-practices/' => 'develop/develop-images',
		56	'engine/tutorials/dockerrepos/' => 'get-started',
		57	'engine/userguide/networking/' => 'network',
		58	'engine/userguide/networking/get-started-overlay/' => 'network',
		59	'engine/reference/commandline/swarm_join_token/' => 'engine/reference/commandline',
		60	'engine/understanding-docker/' => 'engine',
		61	'engine/userguide/dockervolumes/' => 'storage',
		62	'engine/installation/binaries/' => 'install/linux/docker-ce',
		63	'engine/userguide/networking/default_network/dockerlinks/' => 'network',
		64	'engine/reference/api/' => 'develop/sdk',
		65	'engine/admin/systemd/' => 'config/daemon',
		66	'engine/userguide/storagedriver/imagesandcontainers/' => 'storage/storagedriver',
		67	'engine/api/' => 'develop/sdk',
		68	'engine/userguide/networking/get-started-overlay' => 'network',
		69	'engine/userguide/networking/overlay-security-model/' => 'network',
		70	'engine/installation/linux/docker-ce/binaries/' => 'install/linux/docker-ce',
		71	'engine/admin/volumes/' => 'storage/volumes/',
		72	'engine/userguide/networking//' => 'network',
		73	'engine/reference/commandline' => 'engine/reference/commandline/docker',
53	'engine/reference/commandline/' => 'engine/reference/commandline/docker/',	74	'engine/reference/commandline/' => 'engine/reference/commandline/docker/',
54	'engine/reference/api/' => 'engine/api/',	75	}
55	'engine/userguide/dockervolumes/' => 'engine/admin/volumes/volumes/',	76	end
56	'engine/understanding-docker/' => 'engine/docker-overview/',	77
57	'engine/reference/commandline/swarm_join_token/' => 'engine/reference/commandline/swarm_join-token/',	78	version '18' do
58	'engine/api/getting-started/' => 'engine/api/get-started/',	79	self.release = '18.09'
		80	self.base_url = "https://docs.docker.com/v#{release}/"
		81
		82	html_filters.push 'docker/entries', 'docker/clean_html'
		83
		84	options[:container] = '.wrapper .container-fluid .row'
		85
		86	options[:only_patterns] = [/\Aget-started\//, /\Aengine\//, /\Acompose\//, /\Amachine\//, /\Anotary\//]
		87	options[:skip_patterns] = [/\Aengine\/api\/v/, /glossary/, /docker-ee/]
		88
		89	options[:replace_paths] = {
		90	'install/linux/ubuntu/' => 'install/linux/docker-ce',
		91	'get-started/part1' => 'get-started',
		92	'engine/installation/' => 'install',
		93	'engine/installation/linux/linux-postinstall/' => 'install/linux',
		94	'compose/overview/' => 'compose',
		95	'datacenter/install/linux/' => 'ee',
		96	'engine/userguide/' => 'config/daemon',
		97	'engine/admin/' => 'config/daemon',
		98	'opensource/get-help/' => 'opensource',
		99	'engine/tutorials/dockerimages/' => 'get-started',
		100	'engine/admin/volumes/bind-mounts/' => 'storage',
		101	'engine/tutorials/dockervolumes/' => 'storage',
		102	'engine/admin/volumes/volumes/' => 'storage',
		103	'engine/userguide/labels-custom-metadata/' => 'config',
		104	'engine/reference/api/' => 'develop/sdk',
		105	'engine/userguide/eng-image/multistage-build/' => 'develop/develop-images',
		106	'engine/swarm/networking/' => 'network',
		107	'engine/admin/resource_constraints/' => 'config/containers',
		108	'engine/admin/logging/overview/' => 'config/containers/logging',
		109	'engine/userguide/eng-image/dockerfile_best-practices/' => 'develop/develop-images',
		110	'engine/tutorials/dockerrepos/' => 'get-started',
		111	'engine/userguide/networking/' => 'network',
		112	'engine/userguide/networking/get-started-overlay/' => 'network',
		113	'engine/understanding-docker/' => 'engine',
		114	'engine/reference/commandline/swarm_join_token/' => 'engine/reference/commandline',
		115	'engine/userguide/dockervolumes/' => 'storage',
		116	'engine/admin/systemd/' => 'config/daemon',
		117	'engine/userguide/storagedriver/imagesandcontainers/' => 'storage/storagedriver',
		118	'engine/installation/binaries/' => 'install/linux/docker-ce',
		119	'engine/userguide/networking/default_network/dockerlinks/' => 'network',
		120	'engine/userguide/networking/overlay-security-model/' => 'network',
		121	'engine/userguide/networking/get-started-overlay' => 'network',
		122	'engine/api/' => 'develop/sdk',
		123	'engine/installation/linux/docker-ce/binaries/' => 'install/linux/docker-ce',
		124	'engine/admin/volumes/' => 'storage/volumes/',
		125	'engine/userguide/networking//' => 'network',
		126	'engine/reference/commandline' => 'engine/reference/commandline/docker',
		127	'engine/reference/commandline/' => 'engine/reference/commandline/docker/',
		128	}
		129	end
		130
		131	version '17' do
		132	self.release = '17.12'
		133	self.base_url = "https://docs.docker.com/v#{release}/"
		134
		135	html_filters.push 'docker/entries', 'docker/clean_html'
		136
		137	options[:container] = '.wrapper .container-fluid .row'
		138
		139	options[:only_patterns] = [/\Aget-started\//, /\Aengine\//, /\Acompose\//, /\Amachine\//, /\Anotary\//]
		140	options[:skip_patterns] = [/\Aengine\/api\/v/, /glossary/, /docker-ee/]
		141
		142	options[:replace_paths] = {
		143	'get-started/part1' => 'get-started',
		144	'engine/installation/' => 'install',
		145	'engine/installation/linux/linux-postinstall/' => 'install/linux',
		146	'opensource/get-help/' => 'opensource',
		147	'engine/admin/volumes/volumes/' => 'storage',
		148	'engine/tutorials/dockerimages/' => 'get-started',
		149	'engine/admin/volumes/bind-mounts/' => 'storage',
		150	'engine/tutorials/dockervolumes/' => 'storage',
		151	'datacenter/install/aws/' => 'docker-for-aws',
		152	'engine/userguide/' => 'config/daemon',
		153	'engine/admin/' => 'config/daemon',
		154	'engine/userguide/labels-custom-metadata/' => 'config',
		155	'engine/userguide/eng-image/multistage-build/' => 'develop/develop-images',
		156	'engine/swarm/networking/' => 'network',
		157	'engine/admin/resource_constraints/' => 'config/containers',
		158	'engine/admin/logging/overview/' => 'config/containers/logging',
		159	'engine/understanding-docker/' => 'engine',
		160	'engine/userguide/eng-image/dockerfile_best-practices/' => 'develop/develop-images',
		161	'engine/tutorials/dockerrepos/' => 'get-started',
		162	'engine/userguide/networking/' => 'network',
		163	'engine/reference/commandline/swarm_join_token/' => 'edge/engine/reference/commandline',
		164	'engine/userguide/networking/get-started-overlay/' => 'network',
		165	'engine/userguide/dockervolumes/' => 'storage',
		166	'engine/installation/binaries/' => 'install/linux/docker-ce',
		167	'engine/userguide/networking/default_network/dockerlinks/' => 'network',
		168	'engine/reference/api/' => 'develop/sdk',
		169	'engine/admin/live-restore/' => 'config/containers',
		170	'engine/api/' => 'develop/sdk',
		171	'engine/userguide/networking/get-started-overlay' => 'network',
		172	'security/security/' => 'engine/security',
		173	'engine/installation/linux/docker-ce/binaries/' => 'install/linux/docker-ce',
		174	'engine/reference/commandline/' => 'edge/engine/reference/commandline',
		175	'engine/admin/systemd/' => 'config/daemon',
		176	'engine/userguide/storagedriver/imagesandcontainers/' => 'storage/storagedriver',
		177	'engine/userguide/networking/overlay-security-model/' => 'network',
		178	'engine/admin/volumes/' => 'storage/volumes/',
		179	'engine/userguide/networking//' => 'network',
59	}	180	}
60	end	181	end
61		182
@@ -137,5 +258,11 @@ module Docs · 75 unmodified lines
137	options[:container] = '#docs'	258	options[:container] = '#docs'
138	options[:only_patterns] << /\Aswarm\//	259	options[:only_patterns] << /\Aswarm\//
139	end	260	end
		261
		262	def get_latest_version(opts)
		263	doc = fetch_doc('https://docs.docker.com/', opts)
		264	label = doc.at_css('.nav-container button.dropdown-toggle').content.strip
		265	label.scan(/([0-9.]+)/)[0][0]
		266	end
140	end	267	end
141	end	268	end

lib/docs/scrapers/dojo.rb+5−0

       urls.map { |url| "<a href='#{url}'>#{url}</a>" }.join
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://dojotoolkit.org/api/', opts)
+      doc.at_css('#versionSelector > option[selected]').content
+    end
     private
     def get_url_list(json, set = Set.new)

lib/docs/scrapers/drupal.rb+5−0

         /\A[\w\-\.]+\.php\/7\.x\z/
       ]
     end
+    def get_latest_version(opts)
+      json = fetch_json('https://packagist.org/packages/drupal/drupal.json', opts)
+      json['package']['versions'].keys.find {|version| !version.end_with?('-dev')}
+    end
   end
 end

lib/docs/scrapers/electron.rb+5−0

       &copy; 2013&ndash;2018 GitHub Inc.<br>
       Licensed under the MIT license.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://electronjs.org/docs', opts)
+      doc.at_css('.docs-version').content
+    end
   end
 end

lib/docs/scrapers/elixir.rb+33−2

@@ -33,8 +33,34 @@ module Docs · 32 unmodified lines
33	"https://elixir-lang.org/getting-started/introduction.html" ]	33	"https://elixir-lang.org/getting-started/introduction.html" ]
34	end	34	end
35		35
		36	version '1.9' do
		37	self.release = '1.9.1'
		38	self.base_urls = [
		39	"https://hexdocs.pm/elixir/#{release}/",
		40	"https://hexdocs.pm/eex/#{release}/",
		41	"https://hexdocs.pm/ex_unit/#{release}/",
		42	"https://hexdocs.pm/iex/#{release}/",
		43	"https://hexdocs.pm/logger/#{release}/",
		44	"https://hexdocs.pm/mix/#{release}/",
		45	'https://elixir-lang.org/getting-started/'
		46	]
		47	end
		48
		49	version '1.8' do
		50	self.release = '1.8.2'
		51	self.base_urls = [
		52	"https://hexdocs.pm/elixir/#{release}/",
		53	"https://hexdocs.pm/eex/#{release}/",
		54	"https://hexdocs.pm/ex_unit/#{release}/",
		55	"https://hexdocs.pm/iex/#{release}/",
		56	"https://hexdocs.pm/logger/#{release}/",
		57	"https://hexdocs.pm/mix/#{release}/",
		58	'https://elixir-lang.org/getting-started/'
		59	]
		60	end
		61
36	version '1.7' do	62	version '1.7' do
37	self.release = '1.7.3'	63	self.release = '1.7.4'
38	self.base_urls = [	64	self.base_urls = [
39	"https://hexdocs.pm/elixir/#{release}/",	65	"https://hexdocs.pm/elixir/#{release}/",
40	"https://hexdocs.pm/eex/#{release}/",	66	"https://hexdocs.pm/eex/#{release}/",
@@ -47,7 +73,7 @@ module Docs · 6 unmodified lines
47	end	73	end
48		74
49	version '1.6' do	75	version '1.6' do
50	self.release = '1.6.5'	76	self.release = '1.6.6'
51	self.base_urls = [	77	self.base_urls = [
52	"https://hexdocs.pm/elixir/#{release}/",	78	"https://hexdocs.pm/elixir/#{release}/",
53	"https://hexdocs.pm/eex/#{release}/",	79	"https://hexdocs.pm/eex/#{release}/",
@@ -97,5 +123,10 @@ module Docs · 43 unmodified lines
97	'https://elixir-lang.org/getting-started/'	123	'https://elixir-lang.org/getting-started/'
98	]	124	]
99	end	125	end
		126
		127	def get_latest_version(opts)
		128	doc = fetch_doc('https://hexdocs.pm/elixir/api-reference.html', opts)
		129	doc.at_css('h2.sidebar-projectVersion').content.strip[1..-1]
		130	end
100	end	131	end
101	end	132	end

lib/docs/scrapers/ember.rb+5−0

         https://emberjs.com/api/ember-data/2.14/classes/DS
       )
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://emberjs.com/api/ember/release', opts)
+      doc.at_css('.sidebar > .select-container .ember-power-select-selected-item').content.strip
+    end
   end
 end

lib/docs/scrapers/erlang.rb+5−4

     version '21' do
       self.release = '21.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Erlang21'
     end
     version '20' do
       self.release = '20.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Erlang20'
     end
     version '19' do
       self.release = '19.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Erlang19'
     end
     version '18' do
       self.release = '18.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Erlang18'
+    end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://www.erlang.org/downloads', opts)
+      doc.at_css('.col-lg-3 > ul > li').content.strip.sub(/OTP /, '')
     end
   end
 end

lib/docs/scrapers/eslint.rb+4−0

       &copy; JS Foundation and other contributors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('eslint', opts)
+    end
   end
 end

lib/docs/scrapers/express.rb+4−0

       &copy; 2017 StrongLoop, IBM, and other expressjs.com contributors.<br>
       Licensed under the Creative Commons Attribution-ShareAlike License v3.0.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('express', opts)
+    end
   end
 end

lib/docs/scrapers/falcon.rb+5−0

       self.release = '1.2.0'
       self.base_url = "https://falcon.readthedocs.io/en/#{self.release}/"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://falcon.readthedocs.io/en/stable/changes/index.html', opts)
+      doc.at_css('#changelogs ul > li > a').content
+    end
   end
 end

lib/docs/scrapers/fish.rb+11−0

     options[:skip] = %w(design.html license.html)
+    # https://fishshell.com/docs/current/license.html
     options[:attribution] = <<-HTML
       &copy; 2005&ndash;2009 Axel Liljencrantz<br>
       Licensed under the GNU General Public License, version 2.
     HTML
+    version '3.0' do
+      self.release = '3.0.1'
+      self.base_url = "https://fishshell.com/docs/#{version}/"
+    end
     version '2.7' do
       self.release = '2.7.1'
       self.base_url = "https://fishshell.com/docs/#{version}/"
       self.release = '2.2.0'
       self.base_url = "https://fishshell.com/docs/#{version}/"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('http://fishshell.com/docs/current/index.html', opts)
+      doc.at_css('#toc-index').content.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/flow.rb+4−0

       &copy; 2013&ndash;present Facebook Inc.<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('flow-bin', opts)
+    end
   end
 end

lib/docs/scrapers/git.rb+5−0

       &copy; 2005&ndash;2018 Linus Torvalds and others<br>
       Licensed under the GNU General Public License version 2.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://git-scm.com/', opts)
+      doc.at_css('.version').content.strip
+    end
   end
 end

lib/docs/scrapers/gnu/gcc.rb+6−8

     version '7' do
       self.release = '7.3.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcc7'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gcc/"
     end
     version '7 CPP' do
       self.release = '7.3.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcpp7'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/cpp/"
       options[:replace_paths] = CPP_PATHS
     version '6' do
       self.release = '6.4.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcc6'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gcc/"
       options[:root_title] = 'Using the GNU Compiler Collection (GCC)'
     version '6 CPP' do
       self.release = '6.4.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcpp6'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/cpp/"
       options[:replace_paths] = CPP_PATHS
     version '5' do
       self.release = '5.4.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcc5'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gcc/"
       options[:root_title] = 'Using the GNU Compiler Collection (GCC)'
     version '5 CPP' do
       self.release = '5.4.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcpp5'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/cpp/"
       options[:replace_paths] = CPP_PATHS
     version '4' do
       self.release = '4.9.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcc4'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gcc/"
       options[:root_title] = 'Using the GNU Compiler Collection (GCC)'
     version '4 CPP' do
       self.release = '4.9.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gcpp4'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/cpp/"
       options[:replace_paths] = CPP_PATHS
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://gcc.gnu.org/onlinedocs/', opts)
+      label = doc.at_css('ul > li > ul > li > a').content.strip
+      label.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/gnu/gnu_fortran.rb+6−4

     version '7' do
       self.release = '7.3.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gfortran7'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gfortran/"
     end
     version '6' do
       self.release = '6.4.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gfortran6'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gfortran/"
     end
     version '5' do
       self.release = '5.4.0'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gfortran5'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gfortran/"
     end
     version '4' do
       self.release = '4.9.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/gfortran4'
       self.base_url = "https://gcc.gnu.org/onlinedocs/gcc-#{release}/gfortran/"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://gcc.gnu.org/onlinedocs/', opts)
+      label = doc.at_css('ul > li > ul > li > a').content.strip
+      label.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/go.rb+6−1

 module Docs
   class Go < UrlScraper
     self.type = 'go'
-    self.release = '1.11.2'
+    self.release = '1.12'
     self.base_url = 'https://golang.org/pkg/'
     self.links = {
       home: 'https://golang.org/',
       Licensed under the Creative Commons Attribution License 3.0.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://golang.org/project/', opts)
+      doc.at_css('#page ul > li > a').text[3..-1]
+    end
     private
     def parse(response) # Hook here because Nokogori removes whitespace from textareas

lib/docs/scrapers/godot.rb+5−0

       self.release = '2.1'
       self.base_url = "http://docs.godotengine.org/en/#{self.version}/"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.godotengine.org/', opts)
+      doc.at_css('.version').content.strip
+    end
   end
 end

lib/docs/scrapers/graphite.rb+5−0

       &copy; 2011&ndash;2016 The Graphite Project<br>
       Licensed under the Apache License, Version 2.0.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://graphite.readthedocs.io/en/latest/releases.html', opts)
+      doc.at_css('#release-notes li > a').content
+    end
   end
 end

lib/docs/scrapers/grunt.rb+4−0

       &copy; GruntJS Team<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('grunt-cli', opts)
+    end
   end
 end

lib/docs/scrapers/handlebars.rb+4−0

       &copy; 2011&ndash;2017 by Yehuda Katz<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('handlebars', opts)
+    end
   end
 end

lib/docs/scrapers/haskell.rb+8−1

     html_filters.push 'haskell/entries', 'haskell/clean_html'
-    options[:container] = ->(filter) { filter.subpath.start_with?('users_guide') ? '.body' : '#content' }
+    options[:container] = ->(filter) {filter.subpath.start_with?('users_guide') ? '.body' : '#content'}
     options[:only_patterns] = [/\Alibraries\//, /\Ausers_guide\//]
     options[:skip_patterns] = [
       options[:only_patterns] = [/\Alibraries\//]
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://downloads.haskell.org/~ghc/latest/docs/html/', opts)
+      links = doc.css('a').to_a
+      versions = links.map {|link| link['href'].scan(/ghc-([0-9.]+)/)}
+      versions.find {|version| !version.empty?}[0][0]
+    end
   end
 end

lib/docs/scrapers/haxe.rb+6−0

@@ -66,5 +66,11 @@ module Docs · 65 unmodified lines
66	version 'Python' do	66	version 'Python' do
67	self.base_url = 'https://api.haxe.org/python/'	67	self.base_url = 'https://api.haxe.org/python/'
68	end	68	end
		69
		70	def get_latest_version(opts)
		71	doc = fetch_doc('https://api.haxe.org/', opts)
		72	label = doc.at_css('.container.main-content h1 > small').content
		73	label.sub(/version /, '')
		74	end
69	end	75	end
70	end	76	end

lib/docs/scrapers/homebrew.rb+5−1

   class Homebrew < UrlScraper
     self.name = 'Homebrew'
     self.type = 'simple'
-    self.release = '1.8.1'
+    self.release = '2.1.9'
     self.base_url = 'https://docs.brew.sh/'
     self.links = {
       home: 'https://brew.sh',
       &copy; 2009&ndash;present Homebrew contributors<br>
       Licensed under the BSD 2-Clause License.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('Homebrew', 'brew', opts)
+    end
   end
 end

lib/docs/scrapers/http.rb+2−0

@@ -7,6 +7,8 @@ module Docs · 6 unmodified lines
7		7
8	html_filters.push 'http/clean_html', 'http/entries', 'title'	8	html_filters.push 'http/clean_html', 'http/entries', 'title'
9		9
		10	options[:mdn_tag] = 'HTTP'
		11
10	options[:root_title] = 'HTTP'	12	options[:root_title] = 'HTTP'
11	options[:title] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? false : filter.default_title }	13	options[:title] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? false : filter.default_title }
12	options[:container] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? '.content' : nil }	14	options[:container] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? '.content' : nil }

lib/docs/scrapers/immutable.rb+4−0

       JS
       capybara.html
     end
+    def get_latest_version(opts)
+      get_npm_version('immutable', opts)
+    end
   end
 end

lib/docs/scrapers/influxdata.rb+6−0

@@ -46,5 +46,11 @@ module Docs · 45 unmodified lines
46	© 2015 InfluxData, Inc.<br>	46	© 2015 InfluxData, Inc.<br>
47	Licensed under the MIT license.	47	Licensed under the MIT license.
48	HTML	48	HTML
		49
		50	def get_latest_version(opts)
		51	doc = fetch_doc('https://docs.influxdata.com/influxdb/', opts)
		52	label = doc.at_css('.navbar--current-product').content.strip
		53	label.scan(/([0-9.]+)/)[0][0]
		54	end
49	end	55	end
50	end	56	end

lib/docs/scrapers/jasmine.rb+4−0

       &copy; 2008&ndash;2017 Pivotal Labs<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('jasmine', 'jasmine', opts)
+    end
   end
 end

lib/docs/scrapers/jekyll.rb+5−0

       &copy; 2008&ndash;2018 Tom Preston-Werner and Jekyll contributors<br>
       Licensed under the MIT license.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://jekyllrb.com/docs/', opts)
+      doc.at_css('.meta a').content[1..-1]
+    end
   end
 end

lib/docs/scrapers/jest.rb+5−0

       &copy; 2014&ndash;present Facebook Inc.<br>
       Licensed under the BSD License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://jestjs.io/docs/en/getting-started', opts)
+      doc.at_css('header > a > h3').content
+    end
   end
 end

lib/docs/scrapers/jquery/jquery_core.rb+4−0

       /Selectors\/odd/i,
       /index/i
     ]
+    def get_latest_version(opts)
+      get_npm_version('jquery', opts)
+    end
   end
 end

lib/docs/scrapers/jquery/jquery_mobile.rb+5−0

     options[:fix_urls] = ->(url) do
       url.sub! 'http://api.jquerymobile.com/', 'https://api.jquerymobile.com/'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://jquerymobile.com/', opts)
+      doc.at_css('.download-box > .download-option:last-child > span').content.sub(/Version /, '')
+    end
   end
 end

lib/docs/scrapers/jquery/jquery_ui.rb+4−0

     options[:fix_urls] = ->(url) do
       url.sub! 'http://api.jqueryui.com/', 'https://api.jqueryui.com/'
     end
+    def get_latest_version(opts)
+      get_npm_version('jquery-ui', opts)
+    end
   end
 end

lib/docs/scrapers/jsdoc.rb+4−0

       &copy; 2011&ndash;2017 the contributors to the JSDoc 3 documentation project<br>
       Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('jsdoc3', 'jsdoc', opts)
+    end
   end
 end

lib/docs/scrapers/julia.rb+4−0

@@ -49,5 +49,9 @@ module Docs · 48 unmodified lines
49		49
50	html_filters.push 'julia/entries_sphinx', 'julia/clean_html_sphinx', 'sphinx/clean_html'	50	html_filters.push 'julia/entries_sphinx', 'julia/clean_html_sphinx', 'sphinx/clean_html'
51	end	51	end
		52
		53	def get_latest_version(opts)
		54	get_latest_github_release('JuliaLang', 'julia', opts)
		55	end
52	end	56	end
53	end	57	end

lib/docs/scrapers/knockout.rb+4−0

       &copy; Steven Sanderson, the Knockout.js team, and other contributors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('knockout', 'knockout', opts)
+    end
   end
 end

lib/docs/scrapers/koa.rb+4−0

       &copy; 2018 Koa contributors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('koa', opts)
+    end
   end
 end

lib/docs/scrapers/kotlin.rb+15−3

 module Docs
   class Kotlin < UrlScraper
     self.type = 'kotlin'
-    self.release = '1.2.41'
+    self.release = '1.3.41'
     self.base_url = 'https://kotlinlang.org/'
     self.root_path = 'api/latest/jvm/stdlib/index.html'
     self.links = {
       docs/events.html
       docs/resources.html
       docs/reference/grammar.html)
-    options[:replace_paths] = { 'api/latest/jvm/stdlib/' => 'api/latest/jvm/stdlib/index.html' }
+    options[:replace_paths] = {
+      'api/latest/jvm/stdlib/' => 'api/latest/jvm/stdlib/index.html',
+      'docs/reference/coroutines.html' => 'docs/reference/coroutines-overview.html',
+      'api/latest/jvm/stdlib/kotlin/fold.html' => 'api/latest/jvm/stdlib/kotlin.collections/fold.html',
+      'api/latest/jvm/stdlib/kotlin/get-or-else.html' => 'api/latest/jvm/stdlib/kotlin.collections/get-or-else.html',
+      'api/latest/jvm/stdlib/kotlin/map.html' => 'api/latest/jvm/stdlib/kotlin.collections/map.html',
+      'docs/tutorials/native/targeting-multiple-platforms.html' => 'docs/tutorials/native/basic-kotlin-native-app.html',
+      'api/latest/jvm/stdlib/kotlin/-throwable/print-stack-trace.html' => 'api/latest/jvm/stdlib/kotlin/print-stack-trace.html',
+    }
     options[:attribution] = <<-HTML
-      &copy; 2010&ndash;2018 JetBrains s.r.o.<br>
+      &copy; 2010&ndash;2019 JetBrains s.r.o.<br>
       Licensed under the Apache License, Version 2.0.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('JetBrains', 'kotlin', opts)
+    end
   end
 end

lib/docs/scrapers/laravel.rb+4−0

         url
       end
     end
+    def get_latest_version(opts)
+      get_latest_github_release('laravel', 'laravel', opts)
+    end
   end
 end

lib/docs/scrapers/leaflet.rb+16−1

     options[:skip_links] = true
     options[:attribution] = <<-HTML
-      &copy; 2010&ndash;2018 Vladimir Agafonkin<br>
+      &copy; 2010&ndash;2019 Vladimir Agafonkin<br>
       &copy; 2010&ndash;2011, CloudMade<br>
       Maps &copy; OpenStreetMap contributors.
     HTML
+    version '1.5' do
+      self.release = '1.5.0'
+      self.base_url = "https://leafletjs.com/reference-#{release}.html"
+    end
+    version '1.4' do
+      self.release = '1.4.0'
+      self.base_url = "https://leafletjs.com/reference-#{release}.html"
+    end
     version '1.3' do
       self.release = '1.3.4'
       self.base_url = "https://leafletjs.com/reference-#{release}.html"
       self.base_url = "https://leafletjs.com/reference-#{release}.html"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://leafletjs.com/index.html', opts)
+      link = doc.css('ul > li > a').to_a.select {|node| node.content == 'Docs'}.first
+      link['href'].scan(/reference-([0-9.]+)\.html/)[0][0]
+    end
   end
 end

lib/docs/scrapers/less.rb+6−0

@@ -21,5 +21,11 @@ module Docs · 20 unmodified lines
21	© 2009–2016 The Core Less Team<br>	21	© 2009–2016 The Core Less Team<br>
22	Licensed under the Creative Commons Attribution License 3.0.	22	Licensed under the Creative Commons Attribution License 3.0.
23	HTML	23	HTML
		24
		25	def get_latest_version(opts)
		26	doc = fetch_doc('http://lesscss.org/features/', opts)
		27	label = doc.at_css('.footer-links > li').content
		28	label.scan(/([0-9.]+)/)[0][0]
		29	end
24	end	30	end
25	end	31	end

lib/docs/scrapers/liquid.rb+5−0

       &copy; 2005, 2006 Tobias Luetke<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      tags = get_github_tags('Shopify', 'liquid', opts)
+      tags[0]['name'][1..-1]
+    end
   end
 end

lib/docs/scrapers/lodash.rb+5−0

       self.release = '2.4.2'
       self.base_url = "https://lodash.com/docs/#{release}"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://lodash.com/docs/', opts)
+      doc.at_css('#version > option[selected]').content
+    end
   end
 end

lib/docs/scrapers/love.rb+5−0

       &copy; 2006&ndash;2016 L&Ouml;VE Development Team<br>
       Licensed under the GNU Free Documentation License, Version 1.3.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://love2d.org/wiki/Version_History', opts)
+      doc.at_css('#mw-content-text table a').content
+    end
   end
 end

lib/docs/scrapers/lua.rb+5−0

       self.release = '5.1.5'
       self.base_url = 'https://www.lua.org/manual/5.1/'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://www.lua.org/manual/', opts)
+      doc.at_css('p.menubar > a').content
+    end
   end
 end

lib/docs/scrapers/marionette.rb+4−0

@@ -38,5 +38,9 @@ module Docs · 37 unmodified lines
38		38
39	html_filters.push 'marionette/entries_v2'	39	html_filters.push 'marionette/entries_v2'
40	end	40	end
		41
		42	def get_latest_version(opts)
		43	get_npm_version('backbone.marionette', opts)
		44	end
41	end	45	end
42	end	46	end

lib/docs/scrapers/markdown.rb+4−0

       &copy; 2004 John Gruber<br>
       Licensed under the BSD License.
     HTML
+    def get_latest_version(opts)
+      '1.0.0'
+    end
   end
 end

lib/docs/scrapers/matplotlib.rb+13−0

@@ -20,6 +20,15 @@ module Docs · 19 unmodified lines
20	Licensed under the Matplotlib License Agreement.	20	Licensed under the Matplotlib License Agreement.
21	HTML	21	HTML
22		22
		23	version '3.1' do
		24	self.release = '3.1.1'
		25	self.base_urls = [
		26	"https://matplotlib.org/#{release}/api/",
		27	"https://matplotlib.org/#{release}/mpl_toolkits/mplot3d/",
		28	"https://matplotlib.org/#{release}/mpl_toolkits/axes_grid/api/"
		29	]
		30	end
		31
23	version '3.0' do	32	version '3.0' do
24	self.release = '3.0.0'	33	self.release = '3.0.0'
25	self.base_urls = [	34	self.base_urls = [
@@ -64,5 +73,9 @@ module Docs · 38 unmodified lines
64	"https://matplotlib.org/#{release}/mpl_toolkits/axes_grid/api/"	73	"https://matplotlib.org/#{release}/mpl_toolkits/axes_grid/api/"
65	]	74	]
66	end	75	end
		76
		77	def get_latest_version(opts)
		78	get_latest_github_release('matplotlib', 'matplotlib', opts)
		79	end
67	end	80	end
68	end	81	end

lib/docs/scrapers/mdn/css.rb+2−0

@@ -6,6 +6,8 @@ module Docs · 5 unmodified lines
6		6
7	html_filters.push 'css/clean_html', 'css/entries', 'title'	7	html_filters.push 'css/clean_html', 'css/entries', 'title'
8		8
		9	options[:mdn_tag] = 'CSS'
		10
9	options[:root_title] = 'CSS'	11	options[:root_title] = 'CSS'
10		12
11	options[:skip] = %w(/CSS3 /Media/Visual /paged_media /Media/TV /Media/Tactile)	13	options[:skip] = %w(/CSS3 /Media/Visual /paged_media /Media/TV /Media/Tactile)

lib/docs/scrapers/mdn/dom.rb+2−0

@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
8		8
9	html_filters.push 'dom/clean_html', 'dom/entries', 'title'	9	html_filters.push 'dom/clean_html', 'dom/entries', 'title'
10		10
		11	options[:mdn_tag] = 'XSLT_Reference'
		12
11	options[:root_title] = 'DOM'	13	options[:root_title] = 'DOM'
12		14
13	options[:skip] = %w(	15	options[:skip] = %w(

lib/docs/scrapers/mdn/dom_events.rb+2−0

@@ -9,6 +9,8 @@ module Docs · 8 unmodified lines
9	html_filters.insert_after 'clean_html', 'dom_events/clean_html'	9	html_filters.insert_after 'clean_html', 'dom_events/clean_html'
10	html_filters.push 'dom_events/entries', 'title'	10	html_filters.push 'dom_events/entries', 'title'
11		11
		12	options[:mdn_tag] = 'events'
		13
12	options[:root_title] = 'DOM Events'	14	options[:root_title] = 'DOM Events'
13		15
14	options[:skip] = %w(/MozOrientation)	16	options[:skip] = %w(/MozOrientation)

lib/docs/scrapers/mdn/html.rb+2−0

@@ -7,6 +7,8 @@ module Docs · 6 unmodified lines
7		7
8	html_filters.push 'html/clean_html', 'html/entries', 'title'	8	html_filters.push 'html/clean_html', 'html/entries', 'title'
9		9
		10	options[:mdn_tag] = 'HTML'
		11
10	options[:root_title] = 'HTML'	12	options[:root_title] = 'HTML'
11		13
12	options[:title] = ->(filter) do	14	options[:title] = ->(filter) do

lib/docs/scrapers/mdn/javascript.rb+2−0

@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
8		8
9	html_filters.push 'javascript/clean_html', 'javascript/entries', 'title'	9	html_filters.push 'javascript/clean_html', 'javascript/entries', 'title'
10		10
		11	options[:mdn_tag] = 'JavaScript'
		12
11	options[:root_title] = 'JavaScript'	13	options[:root_title] = 'JavaScript'
12		14
13	# Don't want	15	# Don't want

lib/docs/scrapers/mdn/mdn.rb+5−0

       Licensed under the Creative Commons Attribution-ShareAlike License v2.5 or later.
     HTML
+    def get_latest_version(opts)
+      json = fetch_json("https://developer.mozilla.org/en-US/docs/feeds/json/tag/#{options[:mdn_tag]}", opts)
+      DateTime.parse(json[0]['pubdate']).to_time.to_i
+    end
     private
     def process_response?(response)

lib/docs/scrapers/mdn/svg.rb+2−0

@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
8		8
9	html_filters.push 'svg/clean_html', 'svg/entries', 'title'	9	html_filters.push 'svg/clean_html', 'svg/entries', 'title'
10		10
		11	options[:mdn_tag] = 'XSLT_Reference'
		12
11	options[:root_title] = 'SVG'	13	options[:root_title] = 'SVG'
12		14
13	options[:title] = ->(filter) do	15	options[:title] = ->(filter) do

lib/docs/scrapers/mdn/xslt_xpath.rb+2−0

@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
8		8
9	html_filters.push 'xslt_xpath/clean_html', 'xslt_xpath/entries', 'title'	9	html_filters.push 'xslt_xpath/clean_html', 'xslt_xpath/entries', 'title'
10		10
		11	options[:mdn_tag] = 'XSLT_Reference'
		12
11	options[:root_title] = 'XSLT'	13	options[:root_title] = 'XSLT'
12		14
13	options[:only_patterns] = [/\A\/XSLT/, /\A\/XPath/]	15	options[:only_patterns] = [/\A\/XSLT/, /\A\/XPath/]

lib/docs/scrapers/meteor.rb+5−0

       self.base_urls = ['https://guide.meteor.com/v1.3/', "https://docs.meteor.com/v#{self.release}/"]
       options[:fix_urls] = nil
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.meteor.com/#/full/', opts)
+      doc.at_css('select.version-select > option').content
+    end
   end
 end

lib/docs/scrapers/mocha.rb+4−0

       &copy; 2011&ndash;2018 JS Foundation and contributors<br>
       Licensed under the Creative Commons Attribution 4.0 International License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('mocha', opts)
+    end
   end
 end

lib/docs/scrapers/modernizr.rb+4−0

       &copy; 2009&ndash;2017 The Modernizr team<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('modernizr', opts)
+    end
   end
 end

lib/docs/scrapers/moment.rb+5−0

       &copy; JS Foundation and other contributors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('http://momentjs.com/', opts)
+      doc.at_css('.hero-title > h1 > span').content
+    end
   end
 end

lib/docs/scrapers/mongoose.rb+6−0

@@ -26,5 +26,11 @@ module Docs · 25 unmodified lines
26	© 2010 LearnBoost<br>	26	© 2010 LearnBoost<br>
27	Licensed under the MIT License.	27	Licensed under the MIT License.
28	HTML	28	HTML
		29
		30	def get_latest_version(opts)
		31	doc = fetch_doc('https://mongoosejs.com/docs/', opts)
		32	label = doc.at_css('.pure-menu-link').content.strip
		33	label.sub(/Version /, '')
		34	end
29	end	35	end
30	end	36	end

lib/docs/scrapers/nginx.rb+10−4

   class Nginx < UrlScraper
     self.name = 'nginx'
     self.type = 'nginx'
-    self.release = '1.15.5'
+    self.release = '1.17.2'
     self.base_url = 'https://nginx.org/en/docs/'
     self.links = {
       home: 'https://nginx.org/',
-      code: 'http://hg.nginx.org/nginx'
+      code: 'https://hg.nginx.org/nginx'
     }
     html_filters.push 'nginx/clean_html', 'nginx/entries'
     options[:skip_patterns] = [/\/faq\//]
     options[:attribution] = <<-HTML
-      &copy; 2002-2018 Igor Sysoev<br>
+      &copy; 2002-2019 Igor Sysoev<br>
-      &copy; 2011-2018 Nginx, Inc.<br>
+      &copy; 2011-2019 Nginx, Inc.<br>
       Licensed under the BSD License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://nginx.org/en/download.html', opts)
+      table = doc.at_css('#content > table').inner_html
+      table.scan(/nginx-([0-9.]+)</)[0][0]
+    end
   end
 end

lib/docs/scrapers/nginx_lua_module.rb+9−0

     self.slug = 'nginx_lua_module'
     self.release = '0.10.13'
     self.base_url = "https://github.com/openresty/lua-nginx-module/tree/v#{self.release}/"
+    self.links = {
+      code: 'https://github.com/openresty/lua-nginx-module'
+    }
     html_filters.push 'nginx_lua_module/clean_html', 'nginx_lua_module/entries', 'title'
       &copy; 2009&ndash;2018 Yichun "agentzh" Zhang (章亦春), OpenResty Inc.<br>
       Licensed under the BSD License.
     HTML
+    def get_latest_version(opts)
+      tags = get_github_tags('openresty', 'lua-nginx-module', opts)
+      tag = tags.find {|tag| !tag['name'].include?('rc')}
+      tag['name'][1..-1]
+    end
   end
 end

lib/docs/scrapers/nim.rb+5−0

       &copy; 2006&ndash;2018 Andreas Rumpf<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://nim-lang.org/docs/overview.html', opts)
+      doc.at_css('.container > .docinfo > tbody > tr:last-child > td').content.strip
+    end
   end
 end

lib/docs/scrapers/node.rb+7−2

     HTML
     version do
-      self.release = '11.1.0'
+      self.release = '12.8.0'
-      self.base_url = 'https://nodejs.org/dist/latest-v11.x/docs/api/'
+      self.base_url = 'https://nodejs.org/dist/latest-v12.x/docs/api/'
     end
     version '10 LTS' do
       self.release = '4.9.1'
       self.base_url = 'https://nodejs.org/dist/latest-v4.x/docs/api/'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://nodejs.org/en/', opts)
+      doc.at_css('#home-intro > .home-downloadblock:last-of-type > a')['data-version'][1..-1]
+    end
   end
 end

lib/docs/scrapers/nokogiri2.rb+12−4

 module Docs
   class Nokogiri2 < Rdoc
+    # Instructions:
+    #   1. Download the latest release at https://github.com/sparklemotion/nokogiri/releases
+    #   2. Run "bundle install && bundle exec rake docs" (in the Nokogiri directory)
+    #   4. Copy the "doc" directory to "docs/nokgiri"
     self.name = 'Nokogiri'
     self.slug = 'nokogiri'
-    self.release = '1.8.1'
+    self.release = '1.9.0'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/RDoc/Nokogiri'
     html_filters.replace 'rdoc/entries', 'nokogiri2/entries'
     options[:only_patterns] = [/\ANokogiri/, /\AXSD/]
     options[:attribution] = <<-HTML
-      &copy; 2008&ndash;2017 Aaron Patterson, Mike Dalessio, Charles Nutter, Sergio Arbeo<br>
+      &copy; 2008&ndash;2018 Aaron Patterson, Mike Dalessio, Charles Nutter, Sergio Arbeo,<br>
-      Patrick Mahoney, Yoko Harada, Akinori Musha, John Shahid<br>
+      Patrick Mahoney, Yoko Harada, Akinori Musha, John Shahid, Lars Kanis<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('sparklemotion', 'nokogiri', opts)
+    end
   end
 end

lib/docs/scrapers/npm.rb+4−0

       Licensed under the npm License.<br>
       npm is a trademark of npm, Inc.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('npm', 'cli', opts)
+    end
   end
 end

lib/docs/scrapers/numpy.rb+4−1

   class Numpy < FileScraper
     self.name = 'NumPy'
     self.type = 'sphinx'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/numpy/reference/'
     self.root_path = 'index.html'
     self.links = {
       home: 'http://www.numpy.org/',
       self.release = '1.10.4'
       self.base_url = "https://docs.scipy.org/doc/numpy-#{self.release}/reference/"
     end
+    def get_latest_version(opts)
+      get_latest_github_release('numpy', 'numpy', opts)
+    end
   end
 end

lib/docs/scrapers/openjdk.rb+22−3

@@ -1,11 +1,10 @@
1	module Docs	1	module Docs
2	class Openjdk < FileScraper	2	class Openjdk < FileScraper
		3	# Downloaded from packages.debian.org/sid/openjdk-8-doc
		4	# Extracting subdirectory /usr/share/doc/openjdk-8-jre-headless/api
3	self.name = 'OpenJDK'	5	self.name = 'OpenJDK'
4	self.type = 'openjdk'	6	self.type = 'openjdk'
5	self.root_path = 'overview-summary.html'	7	self.root_path = 'overview-summary.html'
6	# Downloaded from packages.debian.org/sid/openjdk-8-doc
7	# Extracting subdirectory /usr/share/doc/openjdk-8-jre-headless/api
8	self.dir = '/Users/Thibaut/DevDocs/Docs/OpenJDK'
9		8
10	html_filters.insert_after 'internal_urls', 'openjdk/clean_urls'	9	html_filters.insert_after 'internal_urls', 'openjdk/clean_urls'
11	html_filters.push 'openjdk/entries', 'openjdk/clean_html'	10	html_filters.push 'openjdk/entries', 'openjdk/clean_html'
@@ -87,5 +86,25 @@ module Docs · 75 unmodified lines
87	def read_file(path)	86	def read_file(path)
88	File.read(path).force_encoding('iso-8859-1').encode('utf-8') rescue nil	87	File.read(path).force_encoding('iso-8859-1').encode('utf-8') rescue nil
89	end	88	end
		89
		90	def get_latest_version(opts)
		91	latest_version = 8
		92	current_attempt = latest_version
		93	attempts = 0
		94
		95	while attempts < 3
		96	current_attempt += 1
		97
		98	doc = fetch_doc("https://packages.debian.org/sid/openjdk-#{current_attempt}-doc", opts)
		99	if doc.at_css('.perror').nil?
		100	latest_version = current_attempt
		101	attempts = 0
		102	else
		103	attempts += 1
		104	end
		105	end
		106
		107	latest_version
		108	end
90	end	109	end
91	end	110	end

lib/docs/scrapers/opentsdb.rb+4−0

       &copy; 2010&ndash;2016 The OpenTSDB Authors<br>
       Licensed under the GNU LGPLv2.1+ and GPLv3+ licenses.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('OpenTSDB', 'opentsdb', opts)
+    end
   end
 end

lib/docs/scrapers/padrino.rb+7−3

   class Padrino < UrlScraper
     self.slug = 'padrino'
     self.type = 'rubydoc'
-    self.release = '0.14.1'
+    self.release = '0.14.4'
-    self.base_url = 'http://www.rubydoc.info/github/padrino/padrino-framework/'
+    self.base_url = 'https://www.rubydoc.info/github/padrino/padrino-framework/'
     self.root_path = 'file/README.rdoc'
     self.initial_paths = %w(index2)
     self.links = {
     options[:container] = ->(filter) { filter.root_page? ? '#filecontents' : '#content' }
     options[:attribution] = <<-HTML
-      &copy; 2010&ndash;2016 Padrino<br>
+      &copy; 2010&ndash;2019 Padrino<br>
       Licensed under the MIT License.
     HTML
     stub 'index2' do
       request_one(url_for('index')).body
     end
+    def get_latest_version(opts)
+      get_github_tags('padrino', 'padrino-framework', opts)[0]['name']
+    end
   end
 end

lib/docs/scrapers/pandas.rb+24−7

     options[:container] = '.document'
     options[:skip] = %w(internals.html release.html contributing.html whatsnew.html)
+    options[:skip_patterns] = [/whatsnew\//]
     options[:attribution] = <<-HTML
       &copy; 2008&ndash;2012, AQR Capital Management, LLC, Lambda Foundry, Inc. and PyData Development Team<br>
       Licensed under the 3-clause BSD License.
     HTML
+    version '0.25' do
+      self.release = '0.25.0'
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+    end
+    version '0.24' do
+      self.release = '0.24.2'
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+    end
     version '0.23' do
       self.release = '0.23.4'
-      self.base_url = "http://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
     end
     version '0.22' do
       self.release = '0.22.0'
-      self.base_url = "http://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
     end
     version '0.21' do
-      self.release = '0.21.0'
+      self.release = '0.21.1'
-      self.base_url = "http://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
     end
     version '0.20' do
       self.release = '0.20.3'
-      self.base_url = "http://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
     end
     version '0.19' do
       self.release = '0.19.2'
-      self.base_url = "http://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
     end
     version '0.18' do
       self.release = '0.18.1'
-      self.base_url = "http://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+      self.base_url = "https://pandas.pydata.org/pandas-docs/version/#{self.release}/"
+    end
+    def get_latest_version(opts)
+      doc = fetch_doc('http://pandas.pydata.org/pandas-docs/stable/', opts)
+      label = doc.at_css('.body > .section > p').content
+      label.scan(/Version: ([0-9.]+)/)[0][0]
     end
   end
 end

lib/docs/scrapers/perl.rb+6−1

   class Perl < FileScraper
     self.name = 'Perl'
     self.type = 'perl'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/Perl'
     self.root_path = 'index.html'
     self.links = {
       home: 'https://www.perl.org/'
       self.release = '5.20.2'
       self.base_url = "https://perldoc.perl.org/#{self.release}/"
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://perldoc.perl.org/', opts)
+      header = doc.at_css('h2.h1').content
+      header.scan(/Perl ([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/phalcon.rb+5−0

       self.release = '2.0.13'
       self.base_url = 'https://docs.phalconphp.com/en/2.0.0/'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.phalconphp.com/', opts)
+      doc.at_css('.header__lang.expand > div > ul > li > a').content
+    end
   end
 end

lib/docs/scrapers/phaser.rb+4−0

       &copy; 2016 Richard Davey, Photon Storm Ltd.<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('photonstorm', 'phaser', opts)
+    end
   end
 end

lib/docs/scrapers/phoenix.rb+5−0

         HTML
       end
     }
+    def get_latest_version(opts)
+      doc = fetch_doc('https://hexdocs.pm/phoenix/Phoenix.html', opts)
+      doc.at_css('.sidebar-projectVersion').content.strip[1..-1]
+    end
   end
 end

lib/docs/scrapers/php.rb+8−3

 module Docs
   class Php < FileScraper
+    # Downloaded from php.net/download-docs.php
     include FixInternalUrlsBehavior
     self.name = 'PHP'
       code: 'https://git.php.net/?p=php-src.git;a=summary'
     }
-    # Downloaded from php.net/download-docs.php
-    self.dir = '/Users/Thibaut/DevDocs/Docs/PHP'
     html_filters.push 'php/internal_urls', 'php/entries', 'php/clean_html', 'title'
     text_filters.push 'php/fix_urls'
       &copy; 1997&ndash;2018 The PHP Documentation Group<br>
       Licensed under the Creative Commons Attribution License v3.0 or later.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://secure.php.net/manual/en/doc.changelog.php', opts)
+      label = doc.at_css('tbody.gen-changelog > tr > td').content
+      label.split(',').last.strip
+    end
   end
 end

lib/docs/scrapers/phpunit.rb+6−0

@@ -37,5 +37,11 @@ module Docs · 36 unmodified lines
37	self.release = '4.8'	37	self.release = '4.8'
38	self.base_url = "https://phpunit.de/manual/#{release}/en/"	38	self.base_url = "https://phpunit.de/manual/#{release}/en/"
39	end	39	end
		40
		41	def get_latest_version(opts)
		42	doc = fetch_doc('https://phpunit.readthedocs.io/', opts)
		43	label = doc.at_css('.rst-current-version').content.strip
		44	label.scan(/v: ([0-9.]+)/)[0][0]
		45	end
40	end	46	end
41	end	47	end

lib/docs/scrapers/postgresql.rb+6−0

@@ -80,5 +80,11 @@ module Docs · 79 unmodified lines
80		80
81	html_filters.insert_before 'postgresql/extract_metadata', 'postgresql/normalize_class_names'	81	html_filters.insert_before 'postgresql/extract_metadata', 'postgresql/normalize_class_names'
82	end	82	end
		83
		84	def get_latest_version(opts)
		85	doc = fetch_doc('https://www.postgresql.org/docs/current/index.html', opts)
		86	label = doc.at_css('#pgContentWrap h1.title').content
		87	label.scan(/([0-9.]+)/)[0][0]
		88	end
83	end	89	end
84	end	90	end

lib/docs/scrapers/pug.rb+4−0

@@ -18,6 +18,10 @@ module Docs · 17 unmodified lines
18	Licensed under the MIT license.	18	Licensed under the MIT license.
19	HTML	19	HTML
20		20
		21	def get_latest_version(opts)
		22	get_npm_version('pug', opts)
		23	end
		24
21	private	25	private
22		26
23	def parse(response) # Hook here because Nokogori removes whitespace from textareas	27	def parse(response) # Hook here because Nokogori removes whitespace from textareas

lib/docs/scrapers/puppeteer.rb+5−0

       &copy; 2017 Google Inc<br>
       Licensed under the Apache License 2.0.
     HTML
+    def get_latest_version(opts)
+      contents = get_github_file_contents('GoogleChrome', 'puppeteer', 'README.md', opts)
+      contents.scan(/\/v([0-9.]+)\/docs\/api\.md/)[0][0]
+    end
   end
 end

lib/docs/scrapers/pygame.rb+7−2

 module Docs
   class Pygame < UrlScraper
     self.type = 'simple'
-    self.release = '1.9.4'
+    self.release = '1.9.6'
+    self.base_url = 'https://www.pygame.org/docs/'
     self.root_path = 'py-modindex.html'
     self.links = {
       home: 'https://www.pygame.org/',
     options[:only_patterns] = [/ref\//]
     options[:attribution] = <<-HTML
-      &copy; Pygame Developpers.<br>
+      &copy; Pygame Developers.<br>
       Licensed under the GNU LGPL License version 2.1.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('pygame', 'pygame', opts)
+    end
   end
 end

lib/docs/scrapers/python.rb+13−12

       Licensed under the PSF License.
     HTML
-    version '3.7' do
+    version '3.7' do # docs.python.org/3.7/download.html
-      self.release = '3.7.0'
+      self.release = '3.7.4'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Python37' # docs.python.org/3.7/download.html
       self.base_url = 'https://docs.python.org/3.7/'
       html_filters.push 'python/entries_v3', 'sphinx/clean_html', 'python/clean_html'
     end
-    version '3.6' do
+    version '3.6' do # docs.python.org/3.6/download.html
-      self.release = '3.6.6'
+      self.release = '3.6.9'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Python36' # docs.python.org/3.6/download.html
       self.base_url = 'https://docs.python.org/3.6/'
       html_filters.push 'python/entries_v3', 'sphinx/clean_html', 'python/clean_html'
     end
-    version '3.5' do
+    version '3.5' do # docs.python.org/3.5/download.html
-      self.release = '3.5.3'
+      self.release = '3.5.7'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Python35' # docs.python.org/3.5/download.html
       self.base_url = 'https://docs.python.org/3.5/'
       html_filters.push 'python/entries_v3', 'sphinx/clean_html', 'python/clean_html'
     end
-    version '2.7' do
+    version '2.7' do # docs.python.org/2.7/download.html
-      self.release = '2.7.13'
+      self.release = '2.7.16'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/Python27' # docs.python.org/2.7/download.html
       self.base_url = 'https://docs.python.org/2.7/'
       html_filters.push 'python/entries_v2', 'sphinx/clean_html', 'python/clean_html'
     end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://docs.python.org/', opts)
+      doc.at_css('.version_switcher_placeholder').content
+    end
   end
 end

lib/docs/scrapers/q.rb+4−0

       &copy; 2009&ndash;2017 Kristopher Michael Kowal<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('q', opts)
+    end
   end
 end

lib/docs/scrapers/qt.rb+17−2

       Licensed under the GNU Free Documentation License, Version 1.3.
     HTML
+    version '5.13' do
+      self.release = '5.13'
+      self.base_url = 'https://doc.qt.io/qt-5.13/'
+    end
+    version '5.12' do
+      self.release = '5.12'
+      self.base_url = 'https://doc.qt.io/qt-5.12/'
+    end
     version '5.11' do
       self.release = '5.11'
-      self.base_url = 'https://doc.qt.io/qt-5/'
+      self.base_url = 'https://doc.qt.io/archives/qt-5.11/'
     end
     version '5.9' do
     version '5.6' do
       self.release = '5.6'
-      self.base_url = 'https://doc.qt.io/qt-5.6/'
+      self.base_url = 'https://doc.qt.io/archives/qt-5.6/'
+    end
+    def get_latest_version(opts)
+      doc = fetch_doc('https://doc.qt.io/qt-5/index.html', opts)
+      doc.at_css('.mainContent h1.title').content.sub(/Qt /, '')
     end
   end
 end

lib/docs/scrapers/ramda.rb+7−3

 module Docs
   class Ramda < UrlScraper
     self.type = 'ramda'
-    self.release = '0.25.0'
+    self.release = '0.26.1'
-    self.base_url = "http://ramdajs.com/#{release}/docs/"
+    self.base_url = "https://ramdajs.com/#{release}/docs/"
     self.links = {
       home: 'http://ramdajs.com/',
       code: 'https://github.com/ramda/ramda/'
       &copy; 2013&ndash;2016 Scott Sauyet and Michael Hurley<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://ramdajs.com/docs/', opts)
+      doc.at_css('.navbar-brand > .version').content[1..-1]
+    end
   end
 end

lib/docs/scrapers/rdoc/minitest.rb+12−2

 module Docs
   class Minitest < Rdoc
+    # Instructions:
+    #   1. Run "gem update rdoc hoe"
+    #   2. Download the source code at https://github.com/seattlerb/minitest
+    #   3. Run "rake docs" (in the Minitest directory)
+    #   4. Copy the "docs" directory to "docs/minitest"
     self.name = 'Ruby / Minitest'
     self.slug = 'minitest'
-    self.release = '5.10.3'
+    self.release = '5.11.3'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/RDoc/Minitest' # rake docs
     self.links = {
       code: 'https://github.com/seattlerb/minitest'
     }
       &copy; Ryan Davis, seattle.rb<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      contents = get_github_file_contents('seattlerb', 'minitest', 'History.rdoc', opts)
+      contents.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/rdoc/rails.rb+14−3

 module Docs
   class Rails < Rdoc
+    # Instructions:
+    #   1. Download a release at https://github.com/rails/rails/releases
+    #   2. Open "railties/lib/rails/api/task.rb" and comment out any code related to sdoc ("configure_sdoc")
+    #   3. Run "bundle install --without db && bundle exec rake rdoc" (in the Rails directory)
+    #   4. Run "cd guides && bundle exec rake guides:generate:html"
+    #   5. Copy the "guides/output" directory to "html/guides"
+    #   6. Copy the "html" directory to "docs/rails~[version]"
     include FixInternalUrlsBehavior
     self.name = 'Ruby on Rails'
     self.slug = 'rails'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/RDoc/Rails'
     self.initial_paths = %w(guides/index.html)
     self.links = {
       home: 'http://rubyonrails.org/',
     end
     version '5.2' do
-      self.release = '5.2.0'
+      self.release = '5.2.2'
     end
     version '5.1' do
     end
     version '4.2' do
-      self.release = '4.2.10'
+      self.release = '4.2.11'
     end
     version '4.1' do
       self.release = '4.1.16'
     end
+    def get_latest_version(opts)
+      get_latest_github_release('rails', 'rails', opts)
+    end
   end
 end

lib/docs/scrapers/rdoc/ruby.rb+19−19

 module Docs
   class Ruby < Rdoc
-    # Generated with:
+    # Instructions:
-    #   ./configure && make html
+    #   1. Download Ruby's source code
-    # or:
+    #   2. Run "./configure && make html" (in the Ruby directory)
-    #   rdoc \
+    #   3. Copy the ".ext/html" directory to "docs/ruby~[version]"
-    #     --root . \
-    #     --page-dir doc \
-    #     --encoding=UTF-8 \
-    #     --visibility=public \
-    #     --format=darkfish \
-    #     --no-line-numbers \
-    #     --op html .
     include FixInternalUrlsBehavior
     HTML
     version '2.5' do
-      self.release = '2.5.0'
+      self.release = '2.5.3'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/RDoc/Ruby25'
     end
     version '2.4' do
-      self.release = '2.4.3'
+      self.release = '2.4.5'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/RDoc/Ruby24'
     end
     version '2.3' do
-      self.release = '2.3.6'
+      self.release = '2.3.8'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/RDoc/Ruby23'
     end
     version '2.2' do
-      self.release = '2.2.9'
+      self.release = '2.2.10'
-      self.dir = '/Users/Thibaut/DevDocs/Docs/RDoc/Ruby22'
+    end
+    def get_latest_version(opts)
+      tags = get_github_tags('ruby', 'ruby', opts)
+      tags.each do |tag|
+        version = tag['name'].gsub(/_/, '.')[1..-1]
+        if !/^([0-9.]+)$/.match(version).nil? && version.count('.') == 2
+          return version
+        end
+      end
     end
   end
 end

lib/docs/scrapers/react.rb+5−0

       &copy; 2013&ndash;present Facebook Inc.<br>
       Licensed under the Creative Commons Attribution 4.0 International Public License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://reactjs.org/docs/getting-started.html', opts)
+      doc.at_css('a[href="/versions"]').content.strip[1..-1]
+    end
   end
 end

lib/docs/scrapers/react_native.rb+5−0

       &copy; 2015&ndash;2018 Facebook Inc.<br>
       Licensed under the Creative Commons Attribution 4.0 International Public License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://facebook.github.io/react-native/docs/getting-started.html', opts)
+      doc.at_css('header > a > h3').content
+    end
   end
 end

lib/docs/scrapers/redis.rb+6−0

@@ -19,5 +19,11 @@ module Docs · 18 unmodified lines
19	© 2009–2018 Salvatore Sanfilippo<br>	19	© 2009–2018 Salvatore Sanfilippo<br>
20	Licensed under the Creative Commons Attribution-ShareAlike License 4.0.	20	Licensed under the Creative Commons Attribution-ShareAlike License 4.0.
21	HTML	21	HTML
		22
		23	def get_latest_version(opts)
		24	body = fetch('http://download.redis.io/redis-stable/00-RELEASENOTES', opts)
		25	body = body.lines[1..-1].join
		26	body.scan(/Redis ([0-9.]+)/)[0][0]
		27	end
22	end	28	end
23	end	29	end

lib/docs/scrapers/redux.rb+4−0

     stub '' do
       request_one('http://redux.js.org/index.html').body
     end
+    def get_latest_version(opts)
+      get_npm_version('redux', opts)
+    end
   end
 end

lib/docs/scrapers/relay.rb+5−0

       &copy; 2013&ndash;present Facebook Inc.<br>
       Licensed under the BSD License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('http://facebook.github.io/relay/en/', opts)
+      doc.at_css('header > a > h3').content[1..-1]
+    end
   end
 end

lib/docs/scrapers/requirejs.rb+4−0

       &copy; jQuery Foundation and other contributors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('requirejs', opts)
+    end
   end
 end

lib/docs/scrapers/rethinkdb.rb+4−0

@@ -58,6 +58,10 @@ module Docs · 57 unmodified lines
58	CODE	58	CODE
59	end	59	end
60		60
		61	def get_latest_version(opts)
		62	get_latest_github_release('rethinkdb', 'rethinkdb', opts)
		63	end
		64
61	private	65	private
62		66
63	def process_response?(response)	67	def process_response?(response)

lib/docs/scrapers/rust.rb+10−1

+# frozen_string_literal: true
 module Docs
   class Rust < UrlScraper
     self.type = 'rust'
-    self.release = '1.29.1'
+    self.release = '1.30.1'
     self.base_url = 'https://doc.rust-lang.org/'
     self.root_path = 'book/second-edition/index.html'
     self.initial_paths = %w(
     options[:fix_urls] = ->(url) do
       url.sub! %r{(#{Rust.base_url}.+/)\z}, '\1index.html'
       url.sub! '/unicode/u_str', '/unicode/str/'
+      url.sub! '/std/std/', '/std/'
       url
     end
       Licensed under the Apache License, Version 2.0 or the MIT license, at your option.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://www.rust-lang.org/', opts)
+      label = doc.at_css('.button-download + p > a').content
+      label.sub(/Version /, '')
+    end
     private
     REDIRECT_RGX = /http-equiv="refresh"/i

lib/docs/scrapers/salt_stack.rb+36−0

@@ -0,0 +1,36 @@
	1	module Docs
	2	# The official documentation website is heavily rate-limited
	3	#
	4	# The documentation can be generated like this (replace 2019.2 with the correct tag):
	5	# $ git clone https://github.com/saltstack/salt.git --branch 2019.2 --depth 1
	6	# $ cd salt/doc
	7	# $ pip install sphinx
	8	# $ make html
	9	#
	10	# The generated html can be found in salt/doc/_build/html
	11	class SaltStack < FileScraper
	12	self.type = 'simple'
	13	self.slug = 'saltstack'
	14	self.release = '2019.2.0'
	15	self.base_url = 'https://docs.saltstack.com/en/latest/'
	16	self.root_path = 'ref/index.html'
	17	self.links = {
	18	home: 'https://www.saltstack.com/',
	19	code: 'https://github.com/saltstack/salt'
	20	}
	21
	22	html_filters.push 'salt_stack/clean_html', 'salt_stack/entries'
	23
	24	options[:only_patterns] = [/all\//]
	25	options[:container] = '.body-content > .section'
	26
	27	options[:attribution] = <<-HTML
	28	© 2019 SaltStack.<br>
	29	Licensed under the Apache License, Version 2.0.
	30	HTML
	31
	32	def get_latest_version(opts)
	33	get_latest_github_release('saltstack', 'salt', opts)
	34	end
	35	end
	36	end

lib/docs/scrapers/sass.rb+4−0

       &copy; 2006&ndash;2016 Hampton Catlin, Nathan Weizenbaum, and Chris Eppstein<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('sass', 'libsass', opts)
+    end
   end
 end

lib/docs/scrapers/scala.rb+61−0

@@ -0,0 +1,61 @@
	1	module Docs
	2	class Scala < FileScraper
	3	self.name = 'Scala'
	4	self.type = 'scala'
	5	self.links = {
	6	home: 'http://www.scala-lang.org/',
	7	code: 'https://github.com/scala/scala'
	8	}
	9
	10	options[:container] = '#content-container'
	11	options[:attribution] = <<-HTML
	12	© 2002-2019 EPFL, with contributions from Lightbend.<br>
	13	Licensed under the Apache License, Version 2.0.
	14	HTML
	15
	16	# https://downloads.lightbend.com/scala/2.13.0/scala-docs-2.13.0.zip
	17	# Extract api/scala-library into docs/scala~2.13_library
	18	version '2.13 Library' do
	19	self.release = '2.13.0'
	20	self.base_url = 'https://www.scala-lang.org/api/2.13.0/'
	21	self.root_path = 'index.html'
	22
	23	html_filters.push 'scala/entries', 'scala/clean_html'
	24	end
	25
	26	# https://downloads.lightbend.com/scala/2.13.0/scala-docs-2.13.0.zip
	27	# Extract api/scala-reflect into docs/scala~2.13_reflection
	28	version '2.13 Reflection' do
	29	self.release = '2.13.0'
	30	self.base_url = 'https://www.scala-lang.org/api/2.13.0/scala-reflect/'
	31	self.root_path = 'index.html'
	32
	33	html_filters.push 'scala/entries', 'scala/clean_html'
	34	end
	35
	36	# https://downloads.lightbend.com/scala/2.12.9/scala-docs-2.12.9.zip
	37	# Extract api/scala-library into docs/scala~2.12_library
	38	version '2.12 Library' do
	39	self.release = '2.12.9'
	40	self.base_url = 'https://www.scala-lang.org/api/2.12.9/'
	41	self.root_path = 'index.html'
	42
	43	html_filters.push 'scala/entries', 'scala/clean_html'
	44	end
	45
	46	# https://downloads.lightbend.com/scala/2.12.9/scala-docs-2.12.9.zip
	47	# Extract api/scala-reflect into docs/scala~2.12_reflection
	48	version '2.12 Reflection' do
	49	self.release = '2.12.9'
	50	self.base_url = 'https://www.scala-lang.org/api/2.12.9/scala-reflect/'
	51	self.root_path = 'index.html'
	52
	53	html_filters.push 'scala/entries', 'scala/clean_html'
	54	end
	55
	56	def get_latest_version(opts)
	57	doc = fetch_doc('https://www.scala-lang.org/api/current/', opts)
	58	doc.at_css('#doc-version').content
	59	end
	60	end
	61	end

lib/docs/scrapers/scikit_image.rb+5−0

       &copy; 2011 the scikit-image team<br>
       Licensed under the BSD 3-clause License.
     HTML
+    def get_latest_version(opts)
+      tags = get_github_tags('scikit-image', 'scikit-image', opts)
+      tags[0]['name'][1..-1]
+    end
   end
 end

lib/docs/scrapers/scikit_learn.rb+4−0

       Licensed under the 3-clause BSD License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://scikit-learn.org/stable/documentation.html', opts)
+      doc.at_css('.body h1').content.scan(/([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/sinon.rb+5−0

       self.release = '1.17.7'
       self.base_url = "https://sinonjs.org/releases/v#{release}/"
     end
+    def get_latest_version(opts)
+      body = fetch('https://sinonjs.org/', opts)
+      body.scan(/\/releases\/v([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/socketio.rb+4−0

       &copy; 2014&ndash;2015 Automattic<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      get_npm_version('socket.io', opts)
+    end
   end
 end

lib/docs/scrapers/sqlite.rb+5−1

     self.name = 'SQLite'
     self.type = 'sqlite'
     self.release = '3.25.2'
-    self.dir = '/Users/Thibaut/DevDocs/Docs/sqlite/'
     self.base_url = 'https://sqlite.org/'
     self.root_path = 'docs.html'
     self.initial_paths = %w(keyword_index.html)
     options[:attribution] = 'SQLite is in the Public Domain.'
+    def get_latest_version(opts)
+      doc = fetch_doc('https://sqlite.org/chronology.html', opts)
+      doc.at_css('#chrontab > tbody > tr > td:last-child > a').content
+    end
     private
     def parse(response)

lib/docs/scrapers/statsmodels.rb+4−0

       Licensed under the 3-clause BSD License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('http://www.statsmodels.org/stable/', opts)
+      doc.at_css('.sphinxsidebarwrapper h3 + p > b').content[1..-1]
+    end
   end
 end

lib/docs/scrapers/support_tables.rb+6−0

@@ -178,5 +178,11 @@ module Docs · 177 unmodified lines
178	</p>	178	</p>
179	</div>	179	</div>
180	HTML	180	HTML
		181
		182	def get_latest_version(opts)
		183	body = fetch('https://feeds.feedburner.com/WhenCanIUse?format=xml', opts)
		184	timestamp = body.scan(/<updated>([^<]+)<\/updated>/)[0][0]
		185	DateTime.parse(timestamp).to_time.to_i
		186	end
181	end	187	end
182	end	188	end

lib/docs/scrapers/symfony.rb+4−0

       self.release = '2.7.35'
       self.base_url = "https://api.symfony.com/#{version}/"
     end
+    def get_latest_version(opts)
+      get_latest_github_release('symfony', 'symfony', opts)
+    end
   end
 end

lib/docs/scrapers/tcl_tk.rb+5−0

     options[:attribution] = <<-HTML
       Licensed under <a href="http://tcl.tk/software/tcltk/license.html">Tcl/Tk terms</a>
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://www.tcl.tk/man/tcl/contents.htm', opts)
+      doc.at_css('h2').content.scan(/Tk([0-9.]+)/)[0][0]
+    end
   end
 end

lib/docs/scrapers/tensorflow.rb+4−0

@@ -56,6 +56,10 @@ module Docs · 55 unmodified lines
56	/\Aextend/]	56	/\Aextend/]
57	end	57	end
58		58
		59	def get_latest_version(opts)
		60	get_latest_github_release('tensorflow', 'tensorflow', opts)
		61	end
		62
59	private	63	private
60		64
61	def parse(response)	65	def parse(response)

lib/docs/scrapers/terraform.rb+5−0

       &copy; 2018 HashiCorp</br>
       Licensed under the MPL 2.0 License.
     HTML
+    def get_latest_version(opts)
+      contents = get_github_file_contents('hashicorp', 'terraform-website', 'content/config.rb', opts)
+      contents.scan(/version\s+=\s+"([0-9.]+)"/)[0][0]
+    end
   end
 end

lib/docs/scrapers/twig.rb+5−0

       self.release = '1.34.3'
       self.base_url = 'https://twig.symfony.com/doc/1.x/'
     end
+    def get_latest_version(opts)
+      tags = get_github_tags('twigphp', 'Twig', opts)
+      tags[0]['name'][1..-1]
+    end
   end
 end

lib/docs/scrapers/typescript.rb+4−0

       &copy; Microsoft and other contributors<br>
       Licensed under the Apache License, Version 2.0.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('Microsoft', 'TypeScript', opts)
+    end
   end
 end

lib/docs/scrapers/underscore.rb+5−0

       &copy; 2009&ndash;2018 Jeremy Ashkenas, DocumentCloud and Investigative Reporters &amp; Editors<br>
       Licensed under the MIT License.
     HTML
+    def get_latest_version(opts)
+      doc = fetch_doc('https://underscorejs.org/', opts)
+      doc.at_css('.version').content[1...-1]
+    end
   end
 end

lib/docs/scrapers/vagrant.rb+5−0

       &copy; 2010&ndash;2018 Mitchell Hashimoto<br>
       Licensed under the MPL 2.0 License.
     HTML
+    def get_latest_version(opts)
+      contents = get_github_file_contents('hashicorp', 'vagrant', 'website/config.rb', opts)
+      contents.scan(/version\s+=\s+"([0-9.]+)"/)[0][0]
+    end
   end
 end

lib/docs/scrapers/vue.rb+4−0

       self.root_path = '/guide/index.html'
       self.initial_paths = %w(/api/index.html)
     end
+    def get_latest_version(opts)
+      get_latest_github_release('vuejs', 'vue', opts)
+    end
   end
 end

lib/docs/scrapers/vue_router.rb+29−0

@@ -0,0 +1,29 @@
	1	module Docs
	2	class VueRouter < UrlScraper
	3	self.name = 'Vue Router'
	4	self.slug = 'vue_router'
	5	self.type = 'simple'
	6	self.release = '3.1.2'
	7	self.base_url = 'https://router.vuejs.org/'
	8	self.links = {
	9	home: 'https://router.vuejs.org',
	10	code: 'https://github.com/vuejs/vue-router'
	11	}
	12
	13	html_filters.push 'vue_router/entries', 'vue_router/clean_html'
	14
	15	options[:skip_patterns] = [
	16	# Other languages
	17	/^(zh\|ja\|ru\|kr\|fr)\//,
	18	]
	19
	20	options[:attribution] = <<-HTML
	21	© 2013–present Evan You<br>
	22	Licensed under the MIT License.
	23	HTML
	24
	25	def get_latest_version(opts)
	26	get_latest_github_release('vuejs', 'vue-router', opts)
	27	end
	28	end
	29	end

lib/docs/scrapers/vulkan.rb+5−0

       Licensed under the Creative Commons Attribution 4.0 International License.<br>
       Vulkan and the Vulkan logo are registered trademarks of the Khronos Group Inc.
     HTML
+    def get_latest_version(opts)
+      tags = get_github_tags('KhronosGroup', 'Vulkan-Docs', opts)
+      tags[0]['name'][1..-1]
+    end
   end
 end

lib/docs/scrapers/webpack.rb+4−0

         Licensed under the MIT License.
       HTML
     end
+    def get_latest_version(opts)
+      get_npm_version('webpack', opts)
+    end
   end
 end

lib/docs/scrapers/wordpress.rb+44−0

@@ -0,0 +1,44 @@
	1	module Docs
	2	class Wordpress < UrlScraper
	3	self.name = 'WordPress'
	4	self.type = 'wordpress'
	5	self.release = '5.2.2'
	6	self.base_url = 'https://developer.wordpress.org/reference/'
	7	self.initial_paths = %w(
	8	functions/
	9	hooks/
	10	classes/
	11	)
	12
	13	self.links = {
	14	home: 'https://wordpress.org/',
	15	code: 'https://github.com/WordPress/WordPress'
	16	}
	17
	18	html_filters.push 'wordpress/entries', 'wordpress/clean_html'
	19
	20	options[:container] = '#content-area'
	21	options[:trailing_slash] = false
	22	options[:only_patterns] = [
	23	/\Afunctions\//,
	24	/\Ahooks\//,
	25	/\Aclasses\//
	26	]
	27
	28	options[:skip_patterns] = [
	29	/\Afunctions\/page\/\d+/,
	30	/\Ahooks\/page\/\d+/,
	31	/\Aclasses\/page\/\d+/
	32	]
	33
	34	options[:attribution] = <<-HTML
	35	© 2003–2019 WordPress Foundation<br>
	36	Licensed under the GNU GPLv2+ License.
	37	HTML
	38
	39	def get_latest_version(opts)
	40	doc = fetch_doc('https://wordpress.org/download/releases/', opts)
	41	doc.at_css('.releases.latest td').content
	42	end
	43	end
	44	end

lib/docs/scrapers/yarn.rb+4−0

       &copy; 2016&ndash;present Yarn Contributors<br>
       Licensed under the BSD License.
     HTML
+    def get_latest_version(opts)
+      get_latest_github_release('yarnpkg', 'yarn', opts)
+    end
   end
 end

lib/docs/scrapers/yii.rb+4−0

@@ -34,5 +34,9 @@ module Docs · 33 unmodified lines
34		34
35	options[:container] = '.grid_9'	35	options[:container] = '.grid_9'
36	end	36	end
		37
		38	def get_latest_version(opts)
		39	get_latest_github_release('yiisoft', 'yii2', opts)
		40	end
37	end	41	end
38	end	42	end

lib/docs/subscribers/requester_subscriber.rb+13−0

@@ -0,0 +1,13 @@
	1	# frozen_string_literal: true
	2
	3	module Docs
	4	class RequesterSubscriber < Subscriber
	5	self.namespace = 'requester'
	6
	7	def handle_response(event)
	8	if event.duration > 10_000
	9	log "WARN: #{format_url event.payload[:url]} was slow to process (#{(event.duration / 1000).round}s)"
	10	end
	11	end
	12	end
	13	end

lib/tasks/assets.thor+3−0

   option :keep, type: :numeric, default: 0, desc: 'Number of old assets to keep'
   option :verbose, type: :boolean
   def compile
+    load 'tasks/sprites.thor'
+    invoke 'sprites:generate', [], :remove_public_icons => true, :verbose => options[:verbose]
     manifest.compile App.assets_compile
     manifest.clean(options[:keep]) if options[:clean]
   end

lib/tasks/docs.thor+74−13

@@ -12,15 +12,29 @@ class DocsCLI < Thor · 11 unmodified lines
12	end	12	end
13		13
14	desc 'list', 'List available documentations'	14	desc 'list', 'List available documentations'
		15	option :packaged, type: :boolean
15	def list	16	def list
16	output = Docs.all.flat_map do \|doc\|	17	if options[:packaged]
17	name = doc.to_s.demodulize.underscore	18	slugs = Dir[File.join(Docs.store_path, '*.tar.gz')].map { \|f\| File.basename(f, '.tar.gz') }
18	if doc.versioned?	19	names = find_docs_by_slugs(slugs).map do \|doc\|
19	doc.versions.map { \|_doc\| "#{name}@#{_doc.version}" }	20	name = if doc.version?
20	else	21	"#{doc.superclass.to_s.demodulize.underscore}@#{doc.version}"
21	name	22	else
		23	doc.to_s.demodulize.underscore
		24	end
22	end	25	end
23	end.join("\n")	26	else
		27	names = Docs.all.flat_map do \|doc\|
		28	name = doc.to_s.demodulize.underscore
		29	if doc.versioned?
		30	doc.versions.map { \|_doc\| "#{name}@#{_doc.version}" }
		31	else
		32	name
		33	end
		34	end
		35	end
		36
		37	output = names.join("\n")
24		38
25	require 'tty-pager'	39	require 'tty-pager'
26	TTY::Pager.new.page(output)	40	TTY::Pager.new.page(output)
@@ -39,7 +53,7 @@ class DocsCLI < Thor · 12 unmodified lines
39	Docs.install_report :store if options[:verbose]	53	Docs.install_report :store if options[:verbose]
40	if options[:debug]	54	if options[:debug]
41	GC.disable	55	GC.disable
42	Docs.install_report :filter, :request	56	Docs.install_report :filter, :request, :doc
43	end	57	end
44		58
45	if Docs.generate_page(name, options[:version], path)	59	if Docs.generate_page(name, options[:version], path)
@@ -62,7 +76,7 @@ class DocsCLI < Thor · 16 unmodified lines
62	Docs.rescue_errors = true	76	Docs.rescue_errors = true
63	Docs.install_report :store if options[:verbose]	77	Docs.install_report :store if options[:verbose]
64	Docs.install_report :scraper if options[:debug]	78	Docs.install_report :scraper if options[:debug]
65	Docs.install_report :progress_bar, :doc, :image if $stdout.tty?	79	Docs.install_report :progress_bar, :doc, :image, :requester if $stdout.tty?
66		80
67	require 'unix_utils' if options[:package]	81	require 'unix_utils' if options[:package]
68		82
@@ -149,15 +163,55 @@ class DocsCLI < Thor · 80 unmodified lines
149	option :dryrun, type: :boolean	163	option :dryrun, type: :boolean
150	option :packaged, type: :boolean	164	option :packaged, type: :boolean
151	def upload(*names)	165	def upload(*names)
152	names = Dir[File.join(Docs.store_path, '*.tar.gz')].map { \|f\| File.basename(f, '.tar.gz') } if options[:packaged]	166	require 'net/sftp'
153	docs = find_docs(names)	167
		168	if options[:packaged]
		169	slugs = Dir[File.join(Docs.store_path, '*.tar.gz')].map { \|f\| File.basename(f, '.tar.gz') }
		170	docs = find_docs_by_slugs(slugs)
		171	else
		172	docs = find_docs(names)
		173	end
		174
154	assert_docs(docs)	175	assert_docs(docs)
		176
		177	# Verify files are present
		178	docs.each do \|doc\|
		179	unless Dir.exists?(File.join(Docs.store_path, doc.path))
		180	puts "ERROR: directory #{File.join(Docs.store_path, doc.path)} not found."
		181	return
		182	end
		183
		184	unless File.exists?(File.join(Docs.store_path, "#{doc.path}.tar.gz"))
		185	puts "ERROR: package for '#{doc.slug}' documentation not found. Run 'thor docs:package #{doc.slug}' to create it."
		186	return
		187	end
		188	end
		189
		190	# Sync files with S3 (used by the web app)
		191	puts '[S3] Begin syncing.'
155	docs.each do \|doc\|	192	docs.each do \|doc\|
156	puts "Syncing #{doc.path}..."	193	puts "[S3] Syncing #{doc.path}..."
157	cmd = "aws s3 sync #{File.join(Docs.store_path, doc.path)} s3://docs.devdocs.io/#{doc.path} --delete"	194	cmd = "aws s3 sync #{File.join(Docs.store_path, doc.path)} s3://devdocs-assets/#{doc.path} --delete --profile devdocs"
158	cmd << ' --dryrun' if options[:dryrun]	195	cmd << ' --dryrun' if options[:dryrun]
159	system(cmd)	196	system(cmd)
160	end	197	end
		198	puts '[S3] Done syncing.'
		199
		200	# Upload packages to dl.devdocs.io (used by the "thor docs:download" command)
		201	puts '[MaxCDN] Begin uploading.'
		202	Net::SFTP.start('ftp.devdocs-dl.devdocs.netdna-cdn.com', ENV['DEVDOCS_DL_USERNAME'], password: ENV['DEVDOCS_DL_PASSWORD']) do \|sftp\|
		203	docs.each do \|doc\|
		204	filename = "#{doc.path}.tar.gz"
		205	print "[MaxCDN] Uploading #{filename}..."
		206	if options[:dryrun]
		207	print "\n"
		208	else
		209	sftp.upload! File.join(Docs.store_path, filename), File.join('', 'public_html', filename)
		210	print " OK\n"
		211	end
		212	end
		213	end
		214	puts '[MaxCDN] Done uploading.'
161	end	215	end
162		216
163	desc 'commit', '[private]'	217	desc 'commit', '[private]'
@@ -222,6 +276,13 @@ class DocsCLI < Thor · 58 unmodified lines
222	end	276	end
223	end	277	end
224		278
		279	def find_docs_by_slugs(slugs)
		280	slugs.flat_map do \|slug\|
		281	slug, version = slug.split(/~/)
		282	Docs.find_by_slug(slug, version)
		283	end
		284	end
		285
225	def assert_docs(docs)	286	def assert_docs(docs)
226	if docs.empty?	287	if docs.empty?
227	puts 'ERROR: called with no arguments.'	288	puts 'ERROR: called with no arguments.'

lib/tasks/sprites.thor+213−0

@@ -0,0 +1,213 @@
	1	class SpritesCLI < Thor
	2	def self.to_s
	3	'Sprites'
	4	end
	5
	6	def initialize(*args)
	7	require 'docs'
	8	require 'chunky_png'
	9	require 'fileutils'
	10	super
	11	end
	12
	13	desc 'generate [--remove-public-icons] [--verbose]', 'Generate the documentation icon spritesheets'
	14	option :remove_public_icons, type: :boolean, desc: 'Remove public/icons after generating the spritesheets'
	15	option :verbose, type: :boolean
	16	def generate
	17	items = get_items
	18	items_with_icons = items.select {\|item\| item[:has_icons]}
	19	items_without_icons = items.select {\|item\| !item[:has_icons]}
	20	icons_per_row = Math.sqrt(items_with_icons.length).ceil
	21
	22	bg_color = get_sidebar_background
	23
	24	items_with_icons.each_with_index do \|item, index\|
	25	item[:row] = (index / icons_per_row).floor
	26	item[:col] = index - item[:row] * icons_per_row
	27
	28	item[:icon_16] = get_icon(item[:path_16], 16)
	29	item[:icon_32] = get_icon(item[:path_32], 32)
	30
	31	item[:dark_icon_fix] = needs_dark_icon_fix(item[:icon_32], bg_color)
	32	end
	33
	34	return unless items_with_icons.length > 0
	35
	36	log_details(items_with_icons, icons_per_row)
	37
	38	generate_spritesheet(16, items_with_icons, 'assets/images/sprites/docs.png') {\|item\| item[:icon_16]}
	39	generate_spritesheet(32, items_with_icons, 'assets/images/sprites/docs@2x.png') {\|item\| item[:icon_32]}
	40
	41	# Add Mongoose's icon details to docs without custom icons
	42	default_item = items_with_icons.find {\|item\| item[:type] == 'mongoose'}
	43	items_without_icons.each do \|item\|
	44	item[:row] = default_item[:row]
	45	item[:col] = default_item[:col]
	46	item[:dark_icon_fix] = default_item[:dark_icon_fix]
	47	end
	48
	49	save_manifest(items, icons_per_row, 'assets/images/sprites/docs.json')
	50
	51	if options[:remove_public_icons]
	52	logger.info('Removing public/icons')
	53	FileUtils.rm_rf('public/icons')
	54	end
	55	end
	56
	57	private
	58
	59	def get_items
	60	items = Docs.all.map do \|doc\|
	61	base_path = "public/icons/docs/#{doc.slug}"
	62	{
	63	:type => doc.slug,
	64	:path_16 => "#{base_path}/16.png",
	65	:path_32 => "#{base_path}/16@2x.png"
	66	}
	67	end
	68
	69	# Checking paths against an array of possible paths is faster than 200+ File.exist? calls
	70	files = Dir.glob('public/icons/docs/*/.png')
	71
	72	items.each do \|item\|
	73	item[:has_icons] = files.include?(item[:path_16]) && files.include?(item[:path_32])
	74	end
	75	end
	76
	77	def get_icon(path, max_size)
	78	icon = ChunkyPNG::Image.from_file(path)
	79
	80	# Check if the icon is too big
	81	# If it is, resize the image without changing the aspect ratio
	82	if icon.width > max_size \|\| icon.height > max_size
	83	ratio = icon.width.to_f / icon.height
	84	new_width = (icon.width >= icon.height ? max_size : max_size * ratio).floor
	85	new_height = (icon.width >= icon.height ? max_size / ratio : max_size).floor
	86
	87	logger.warn("Icon #{path} is too big: max size is #{max_size} x #{max_size}, icon is #{icon.width} x #{icon.height}, resizing to #{new_width} x #{new_height}")
	88
	89	icon.resample_nearest_neighbor!(new_width, new_height)
	90	end
	91
	92	icon
	93	end
	94
	95	def get_sidebar_background
	96	# This is a hacky way to get the background color of the sidebar
	97	# Unfortunately, it's not possible to get the value of a SCSS variable from a Thor task
	98	# Because hard-coding the value is even worse, we extract it using some regex
	99	path = 'assets/stylesheets/global/_variables-dark.scss'
	100	regex = /--sidebarBackground:\s+([^;]+);/
	101	ChunkyPNG::Color.parse(File.read(path)[regex, 1])
	102	end
	103
	104	def needs_dark_icon_fix(icon, bg_color)
	105	# Determine whether the icon needs to be grayscaled if the user has enabled the dark theme
	106	# The logic is roughly based on https://www.w3.org/TR/2008/REC-WCAG20-20081211/#visual-audio-contrast
	107	contrast = icon.pixels.select {\|pixel\| ChunkyPNG::Color.a(pixel) > 0}.map do \|pixel\|
	108	get_contrast(bg_color, pixel)
	109	end
	110
	111	avg = contrast.reduce(:+) / contrast.size.to_f
	112	avg < 3.5
	113	end
	114
	115	def get_contrast(base, other)
	116	# Calculating the contrast ratio as described in the WCAG 2.0:
	117	# https://www.w3.org/TR/2008/REC-WCAG20-20081211/#contrast-ratiodef
	118	l1 = get_luminance(base) + 0.05
	119	l2 = get_luminance(other) + 0.05
	120	ratio = l1 / l2
	121	l2 > l1 ? 1 / ratio : ratio
	122	end
	123
	124	def get_luminance(color)
	125	rgb = [
	126	ChunkyPNG::Color.r(color).to_f,
	127	ChunkyPNG::Color.g(color).to_f,
	128	ChunkyPNG::Color.b(color).to_f
	129	]
	130
	131	# Calculating the relative luminance as described in the WCAG 2.0:
	132	# https://www.w3.org/TR/2008/REC-WCAG20-20081211/#relativeluminancedef
	133
	134	rgb.map! do \|value\|
	135	value /= 255
	136	value < 0.03928 ? value / 12.92 : ((value + 0.055) / 1.055) ** 2.4
	137	end
	138
	139	0.2126 * rgb[0] + 0.7152 * rgb[1] + 0.0722 * rgb[2]
	140	end
	141
	142	def generate_spritesheet(size, items_with_icons, output_path, &item_to_icon)
	143	logger.info("Generating spritesheet #{output_path} with icons of size #{size} x #{size}")
	144
	145	icons_per_row = Math.sqrt(items_with_icons.length).ceil
	146	spritesheet = ChunkyPNG::Image.new(size * icons_per_row, size * icons_per_row)
	147
	148	items_with_icons.each do \|item\|
	149	icon = item_to_icon.call(item)
	150
	151	# Calculate the base coordinates
	152	base_x = item[:col] * size
	153	base_y = item[:row] * size
	154
	155	# Center the icon if it's not a perfect rectangle
	156	x = base_x + ((size - icon.width) / 2).floor
	157	y = base_y + ((size - icon.height) / 2).floor
	158
	159	spritesheet.compose!(icon, x, y)
	160	end
	161
	162	FileUtils.mkdir_p(File.dirname(output_path))
	163	spritesheet.save(output_path)
	164	end
	165
	166	def save_manifest(items, icons_per_row, path)
	167	logger.info("Saving spritesheet details to #{path}")
	168
	169	FileUtils.mkdir_p(File.dirname(path))
	170
	171	# Only save the details that the scss file needs
	172	manifest_items = items.map do \|item\|
	173	{
	174	:type => item[:type],
	175	:row => item[:row],
	176	:col => item[:col],
	177	:dark_icon_fix => item[:dark_icon_fix]
	178	}
	179	end
	180
	181	manifest = {:icons_per_row => icons_per_row, :items => manifest_items}
	182
	183	File.open(path, 'w') do \|f\|
	184	f.write(JSON.generate(manifest))
	185	end
	186	end
	187
	188	def log_details(items_with_icons, icons_per_row)
	189	logger.debug("Amount of icons: #{items_with_icons.length}")
	190	logger.debug("Amount of icons needing the dark icon fix: #{items_with_icons.count {\|item\| item[:dark_icon_fix]}}")
	191	logger.debug("Amount of icons per row: #{icons_per_row}")
	192
	193	max_type_length = items_with_icons.map {\|item\| item[:type].length}.max
	194	border = "+#{'-' * (max_type_length + 2)}+#{'-' * 5}+#{'-' * 8}+#{'-' * 15}+"
	195
	196	logger.debug(border)
	197	logger.debug("\| #{'Type'.ljust(max_type_length)} \| Row \| Column \| Dark icon fix \|")
	198	logger.debug(border)
	199
	200	items_with_icons.each do \|item\|
	201	logger.debug("\| #{item[:type].ljust(max_type_length)} \| #{item[:row].to_s.ljust(3)} \| #{item[:col].to_s.ljust(6)} \| #{(item[:dark_icon_fix] ? 'Yes' : 'No').ljust(13)} \|")
	202	end
	203
	204	logger.debug(border)
	205	end
	206
	207	def logger
	208	@logger \|\|= Logger.new($stdout).tap do \|logger\|
	209	logger.level = options[:verbose] ? Logger::DEBUG : Logger::INFO
	210	logger.formatter = proc {\|severity, datetime, progname, msg\| "#{msg}\n"}
	211	end
	212	end
	213	end

lib/tasks/updates.thor+315−0

@@ -0,0 +1,315 @@
	1	class UpdatesCLI < Thor
	2	# The GitHub user that is allowed to upload reports
	3	UPLOAD_USER = 'devdocs-bot'
	4
	5	# The repository to create an issue in when uploading the results
	6	UPLOAD_REPO = 'freeCodeCamp/devdocs'
	7
	8	def self.to_s
	9	'Updates'
	10	end
	11
	12	def initialize(*args)
	13	require 'docs'
	14	require 'progress_bar'
	15	require 'terminal-table'
	16	require 'date'
	17	super
	18	end
	19
	20	desc 'check [--github-token] [--upload] [--verbose] [doc]...', 'Check for outdated documentations'
	21	option :github_token, :type => :string
	22	option :upload, :type => :boolean
	23	option :verbose, :type => :boolean
	24	def check(*names)
	25	# Convert names to a list of Scraper instances
	26	# Versions are omitted, if v10 is outdated than v8 is aswell
	27	docs = names.map {\|name\| Docs.find(name.split(/@\|~/)[0], false)}.uniq
	28
	29	# Check all documentations for updates when no arguments are given
	30	docs = Docs.all if docs.empty?
	31
	32	opts = {
	33	logger: logger
	34	}
	35
	36	if options.key?(:github_token)
	37	opts[:github_token] = options[:github_token]
	38	end
	39
	40	with_progress_bar do \|bar\|
	41	bar.max = docs.length
	42	bar.write
	43	end
	44
	45	results = docs.map do \|doc\|
	46	result = check_doc(doc, opts)
	47	with_progress_bar(&:increment!)
	48	result
	49	end
	50
	51	process_results(results)
	52	rescue Docs::DocNotFound => error
	53	logger.error(error)
	54	logger.info('Run "thor docs:list" to see the list of docs.')
	55	end
	56
	57	private
	58
	59	def check_doc(doc, opts)
	60	logger.debug("Checking #{doc.name}")
	61
	62	instance = doc.versions.first.new
	63	scraper_version = instance.get_scraper_version(opts)
	64	latest_version = instance.get_latest_version(opts)
	65
	66	{
	67	name: doc.name,
	68	scraper_version: format_version(scraper_version),
	69	latest_version: format_version(latest_version),
	70	is_outdated: instance.is_outdated(scraper_version, latest_version)
	71	}
	72	rescue NotImplementedError
	73	logger.warn("Couldn't check #{doc.name}, get_latest_version is not implemented")
	74	error_result(doc, '`get_latest_version` is not implemented')
	75	rescue => error
	76	logger.error("Error while checking #{doc.name}\n#{error.full_message.strip}")
	77	error_result(doc, error.message.gsub(/'/, '`'))
	78	end
	79
	80	def format_version(version)
	81	str = version.to_s
	82
	83	# If the version is numeric and greater than or equal to 1e9 it's probably a timestamp
	84	return str if str.match(/^(\d)+$/).nil? or str.to_i < 1e9
	85
	86	DateTime.strptime(str, '%s').strftime('%F')
	87	end
	88
	89	def error_result(doc, reason)
	90	{
	91	name: doc.name,
	92	error: reason
	93	}
	94	end
	95
	96	def process_results(results)
	97	successful_results = results.select {\|result\| result.key?(:is_outdated)}
	98	failed_results = results.select {\|result\| result.key?(:error)}
	99
	100	up_to_date_results = successful_results.select {\|result\| !result[:is_outdated]}
	101	outdated_results = successful_results.select {\|result\| result[:is_outdated]}
	102
	103	log_results(outdated_results, up_to_date_results, failed_results)
	104	upload_results(outdated_results, up_to_date_results, failed_results) if options[:upload]
	105	end
	106
	107	#
	108	# Result logging methods
	109	#
	110
	111	def log_results(outdated_results, up_to_date_results, failed_results)
	112	log_failed_results(failed_results) unless failed_results.empty?
	113	log_successful_results('Up-to-date', up_to_date_results) unless up_to_date_results.empty?
	114	log_successful_results('Outdated', outdated_results) unless outdated_results.empty?
	115	end
	116
	117	def log_successful_results(label, results)
	118	title = "#{label} documentations (#{results.length})"
	119	headings = ['Documentation', 'Scraper version', 'Latest version']
	120	rows = results.map {\|result\| [result[:name], result[:scraper_version], result[:latest_version]]}
	121
	122	table = Terminal::Table.new :title => title, :headings => headings, :rows => rows
	123	puts table
	124	end
	125
	126	def log_failed_results(results)
	127	title = "Documentations that could not be checked (#{results.length})"
	128	headings = %w(Documentation Reason)
	129	rows = results.map {\|result\| [result[:name], result[:error]]}
	130
	131	table = Terminal::Table.new :title => title, :headings => headings, :rows => rows
	132	puts table
	133	end
	134
	135	#
	136	# Upload methods
	137	#
	138
	139	def upload_results(outdated_results, up_to_date_results, failed_results)
	140	# We can't create issues without a GitHub token
	141	unless options.key?(:github_token)
	142	logger.error("Please specify a GitHub token with the public_repo permission for #{UPLOAD_USER} with the --github-token parameter")
	143	return
	144	end
	145
	146	logger.info('Uploading the results to a new GitHub issue')
	147
	148	logger.info('Checking if the GitHub token belongs to the correct user')
	149	user = github_get('/user')
	150
	151	# Only allow the DevDocs bot to upload reports
	152	unless user['login'] == UPLOAD_USER
	153	logger.error("Only #{UPLOAD_USER} is supposed to upload the results to a new issue. The specified github token is not for #{UPLOAD_USER}.")
	154	return
	155	end
	156
	157	logger.info('Creating a new GitHub issue')
	158
	159	issue = results_to_issue(outdated_results, up_to_date_results, failed_results)
	160	created_issue = github_post("/repos/#{UPLOAD_REPO}/issues", issue)
	161
	162	logger.info('Checking if the previous issue is still open')
	163
	164	search_params = {
	165	q: "Documentation versions report in:title author:#{UPLOAD_USER} is:issue repo:#{UPLOAD_REPO}",
	166	sort: 'created',
	167	order: 'desc'
	168	}
	169
	170	matching_issues = github_get('/search/issues', search_params)
	171	previous_issue = matching_issues['items'].find {\|item\| item['number'] != created_issue['number']}
	172
	173	if previous_issue.nil?
	174	logger.info('No previous issue found')
	175	log_upload_success(created_issue)
	176	else
	177	logger.info('Commenting on the previous issue')
	178
	179	comment = "This report was superseded by ##{created_issue['number']}."
	180	github_post("/repos/#{UPLOAD_REPO}/issues/#{previous_issue['number']}/comments", {body: comment})
	181	if previous_issue['closed_at'].nil?
	182	logger.info('Closing the previous issue')
	183	github_patch("/repos/#{UPLOAD_REPO}/issues/#{previous_issue['number']}", {state: 'closed'})
	184	log_upload_success(created_issue)
	185	else
	186	logger.info('The previous issue has already been closed')
	187	log_upload_success(created_issue)
	188	end
	189	end
	190	end
	191
	192	def results_to_issue(outdated_results, up_to_date_results, failed_results)
	193	results = [
	194	successful_results_to_markdown('Outdated', outdated_results),
	195	successful_results_to_markdown('Up-to-date', up_to_date_results),
	196	failed_results_to_markdown(failed_results)
	197	]
	198
	199	results_str = results.select {\|result\| !result.nil?}.join("\n\n")
	200	travis_str = ENV['TRAVIS'].nil? ? '' : "\n\nThis issue was created by Travis CI build [##{ENV['TRAVIS_BUILD_NUMBER']}](#{ENV['TRAVIS_BUILD_WEB_URL']})."
	201
	202	title = "Documentation versions report for #{Date.today.strftime('%B %Y')}"
	203	body = <<-MARKDOWN
	204	## What is this?
	205
	206	This is an automatically created issue which contains information about the version status of the documentations available on DevDocs. The results of this report can be used by maintainers when updating outdated documentations.
	207
	208	Maintainers can close this issue when all documentations are up-to-date. The issue is also automatically closed when the next report is created.#{travis_str}
	209
	210	## Results
	211
	212	The #{outdated_results.length + up_to_date_results.length + failed_results.length} documentations are divided as follows:
	213	- #{outdated_results.length} that #{outdated_results.length == 1 ? 'is' : 'are'} outdated
	214	- #{up_to_date_results.length} that #{up_to_date_results.length == 1 ? 'is' : 'are'} up-to-date (patch updates are ignored)
	215	- #{failed_results.length} that could not be checked
	216	MARKDOWN
	217
	218	{
	219	title: title,
	220	body: body.strip + "\n\n" + results_str
	221	}
	222	end
	223
	224	def successful_results_to_markdown(label, results)
	225	return nil if results.empty?
	226
	227	title = "#{label} documentations (#{results.length})"
	228	headings = ['Documentation', 'Scraper version', 'Latest version']
	229	rows = results.map {\|result\| [result[:name], result[:scraper_version], result[:latest_version]]}
	230
	231	results_to_markdown(title, headings, rows)
	232	end
	233
	234	def failed_results_to_markdown(results)
	235	return nil if results.empty?
	236
	237	title = "Documentations that could not be checked (#{results.length})"
	238	headings = %w(Documentation Reason)
	239	rows = results.map {\|result\| [result[:name], result[:error]]}
	240
	241	results_to_markdown(title, headings, rows)
	242	end
	243
	244	def results_to_markdown(title, headings, rows)
	245	"<details>\n<summary>#{title}</summary>\n\n#{create_markdown_table(headings, rows)}\n</details>"
	246	end
	247
	248	def create_markdown_table(headings, rows)
	249	header = headings.join(' \| ')
	250	separator = '-\|' * headings.length
	251	body = rows.map {\|row\| row.join(' \| ')}
	252
	253	header + "\n" + separator[0...-1] + "\n" + body.join("\n")
	254	end
	255
	256	def log_upload_success(created_issue)
	257	logger.info("Successfully uploaded the results to #{created_issue['html_url']}")
	258	end
	259
	260	#
	261	# HTTP utilities
	262	#
	263
	264	def github_get(endpoint, params = {})
	265	github_request(endpoint, {method: :get, params: params})
	266	end
	267
	268	def github_post(endpoint, params)
	269	github_request(endpoint, {method: :post, body: params.to_json})
	270	end
	271
	272	def github_patch(endpoint, params)
	273	github_request(endpoint, {method: :patch, body: params.to_json})
	274	end
	275
	276	def github_request(endpoint, opts)
	277	url = "https://api.github.com#{endpoint}"
	278
	279	# GitHub token authentication
	280	opts[:headers] = {
	281	Authorization: "token #{options[:github_token]}"
	282	}
	283
	284	# GitHub requires the Content-Type to be application/json when a body is passed
	285	if opts.key?(:body)
	286	opts[:headers]['Content-Type'] = 'application/json'
	287	end
	288
	289	logger.debug("Making a #{opts[:method]} request to #{url}")
	290	response = Docs::Request.run(url, opts)
	291
	292	# response.success? is false if the response code is 201
	293	# GitHub returns 201 Created after an issue is created
	294	if response.success? \|\| response.code == 201
	295	JSON.parse(response.body)
	296	else
	297	logger.error("Couldn't make a #{opts[:method]} request to #{url} (response code #{response.code})")
	298	nil
	299	end
	300	end
	301
	302	# A utility method which ensures no progress bar is shown when stdout is not a tty
	303	def with_progress_bar(&block)
	304	return unless $stdout.tty?
	305	@progress_bar \|\|= ::ProgressBar.new
	306	block.call @progress_bar
	307	end
	308
	309	def logger
	310	@logger \|\|= Logger.new($stdout).tap do \|logger\|
	311	logger.level = options[:verbose] ? Logger::DEBUG : Logger::INFO
	312	logger.formatter = proc {\|severity, datetime, progname, msg\| "[#{severity}] #{msg}\n"}
	313	end
	314	end
	315	end

public/docs/docs.json+0−1

@@ -1 +0,0 @@
1	[]

public/favicon.pxm+0−0

Binary file not shown.

public/favicon@2x.pxm+0−0

Binary file not shown.

public/icons/docs-1.pxm+0−0

Binary file not shown.

public/icons/docs-1@2x.pxm+0−0

Binary file not shown.

public/icons/docs-2.pxm+0−0

Binary file not shown.

public/icons/docs-2@2x.pxm+0−0

Binary file not shown.

public/icons/docs/bluebird/16@2x.png+0−0

Binary file not shown.

public/icons/docs/composer/16.png+0−0

Binary file not shown.

public/icons/docs/composer/16@2x.png+0−0

Binary file not shown.

public/icons/docs/composer/SOURCE+1−0

@@ -0,0 +1 @@
		1	https://github.com/composer/getcomposer.org/blob/master/web/img/logo-composer-transparent.png

public/icons/docs/css/CSS3.sketch/Data+0−0

Binary file not shown.

public/icons/docs/css/CSS3.sketch/QuickLook/Preview.png+0−0

Binary file not shown.

public/icons/docs/css/CSS3.sketch/QuickLook/Thumbnail.png+0−0

Binary file not shown.

public/icons/docs/css/CSS3.sketch/fonts+0−0

Empty file added.

public/icons/docs/css/CSS3.sketch/version+1−0

@@ -0,0 +1 @@
		1	10

public/icons/docs/cypress/16.png+0−0

Binary file not shown.

public/icons/docs/cypress/16@2x.png+0−0

Binary file not shown.

public/icons/docs/cypress/SOURCE+1−0

@@ -0,0 +1 @@
		1	https://github.com/cypress-io/cypress-documentation/raw/develop/themes/cypress/source/img/favicon.ico

public/icons/docs/dom/DOM.sketch/Data+0−0

Binary file not shown.

public/icons/docs/dom/DOM.sketch/QuickLook/Preview.png+0−0

Binary file not shown.

public/icons/docs/dom/DOM.sketch/QuickLook/Thumbnail.png+0−0

Binary file not shown.

public/icons/docs/dom/DOM.sketch/fonts+0−0

Empty file added.

public/icons/docs/dom/DOM.sketch/version+1−0

@@ -0,0 +1 @@
		1	10

public/icons/docs/dom_events/DOM_events.sketch/Data+0−0

Binary file not shown.

public/icons/docs/dom_events/DOM_events.sketch/QuickLook/Preview.png+0−0

Binary file not shown.

public/icons/docs/dom_events/DOM_events.sketch/QuickLook/Thumbnail.png+0−0

Binary file not shown.

public/icons/docs/dom_events/DOM_events.sketch/fonts+0−0

Empty file added.

public/icons/docs/dom_events/DOM_events.sketch/version+1−0

@@ -0,0 +1 @@
		1	10

public/icons/docs/express/16.pxm+0−0

Binary file not shown.

public/icons/docs/express/16@2x.pxm+0−0

Binary file not shown.

public/icons/docs/html/HTML5.sketch/Data+0−0

Binary file not shown.

public/icons/docs/html/HTML5.sketch/QuickLook/Preview.png+0−0

Binary file not shown.

public/icons/docs/html/HTML5.sketch/QuickLook/Thumbnail.png+0−0

Binary file not shown.

public/icons/docs/html/HTML5.sketch/fonts+0−0

Empty file added.

public/icons/docs/html/HTML5.sketch/version+1−0

@@ -0,0 +1 @@
		1	10

public/icons/docs/http/16.pxm+0−0

Binary file not shown.

public/icons/docs/http/16@2x.pxm+0−0

Binary file not shown.

public/icons/docs/javascript/16.pxm+0−0

Binary file not shown.

public/icons/docs/javascript/16@2x.pxm+0−0

Binary file not shown.

public/icons/docs/knockout/16@2x.pxm+0−0

Binary file not shown.

public/icons/docs/less/less.pxm+0−0

Binary file not shown.

public/icons/docs/moment/moment.sketch/Data+0−0

Binary file not shown.

public/icons/docs/moment/moment.sketch/QuickLook/Preview.png+0−0

Binary file not shown.

public/icons/docs/moment/moment.sketch/QuickLook/Thumbnail.png+0−0

Binary file not shown.

public/icons/docs/moment/moment.sketch/fonts+0−0

Empty file added.

public/icons/docs/moment/moment.sketch/metadata+14−0

@@ -0,0 +1,14 @@
	1	<?xml version="1.0" encoding="UTF-8"?>
	2	<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
	3	<plist version="1.0">
	4	<dict>
	5	<key>app</key>
	6	<string>com.bohemiancoding.sketch</string>
	7	<key>build</key>
	8	<integer>5302</integer>
	9	<key>commit</key>
	10	<string>9460a4bc62af5e9ba50dd4143578fd9401710ce5</string>
	11	<key>version</key>
	12	<integer>18</integer>
	13	</dict>
	14	</plist>

public/icons/docs/moment/moment.sketch/version+1−0

@@ -0,0 +1 @@
		1	18

public/icons/docs/nokogiri/icon.pxm+0−0

Binary file not shown.

public/icons/docs/php/16@2x.pxm+0−0

Binary file not shown.

public/icons/docs/rails/rails.pxm+0−0

Binary file not shown.

public/icons/docs/saltstack/16.png+0−0

Binary file not shown.

public/icons/docs/saltstack/16@2x.png+0−0

Binary file not shown.

public/icons/docs/saltstack/SOURCE+1−0

@@ -0,0 +1 @@
		1	https://github.com/saltstack/salt/blob/develop/doc/_static/salt-logo.svg

public/icons/docs/sass/sass.pxm+0−0

Binary file not shown.

public/icons/docs/scala/16.png+0−0

Binary file not shown.

public/icons/docs/scala/16@2x.png+0−0

Binary file not shown.

public/icons/docs/support_tables/16.pxm+0−0

Binary file not shown.

public/icons/docs/support_tables/16@2x.pxm+0−0

Binary file not shown.

public/icons/docs/vue_router/16.png+0−0

Binary file not shown.

public/icons/docs/vue_router/16@2x.png+0−0

Binary file not shown.

public/icons/docs/vue_router/SOURCE+1−0

@@ -0,0 +1 @@
		1	https://github.com/vuejs/vuejs.org/blob/master/assets/logo.ai

public/icons/docs/wordpress/16.png+0−0

Binary file not shown.

public/icons/docs/wordpress/16@2x.png+0−0

Binary file not shown.

public/icons/docs/wordpress/SOURCE+1−0

@@ -0,0 +1 @@
		1	https://wordpress.org/about/logos/

public/icons/docs/xpath/XPath.sketch/Data+0−0

Binary file not shown.

public/icons/docs/xpath/XPath.sketch/QuickLook/Preview.png+0−0

Binary file not shown.

public/icons/docs/xpath/XPath.sketch/QuickLook/Thumbnail.png+0−0

Binary file not shown.

public/icons/docs/xpath/XPath.sketch/metadata+18−0

@@ -0,0 +1,18 @@
	1	<?xml version="1.0" encoding="UTF-8"?>
	2	<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
	3	<plist version="1.0">
	4	<dict>
	5	<key>app</key>
	6	<string>com.bohemiancoding.sketch</string>
	7	<key>build</key>
	8	<integer>5355</integer>
	9	<key>commit</key>
	10	<string>b7d299b0a34651d1a0e066786b75aa36168d5809</string>
	11	<key>fonts</key>
	12	<array/>
	13	<key>length</key>
	14	<integer>22373</integer>
	15	<key>version</key>
	16	<integer>18</integer>
	17	</dict>
	18	</plist>

public/icons/docs/xpath/XPath.sketch/version+1−0

@@ -0,0 +1 @@
		1	18

public/images/apple-icon.pxm+0−0

Binary file not shown.

public/images/fluid-icon.pxm+0−0

Binary file not shown.

public/images/icon.pxm+0−0

Binary file not shown.

public/images/webapp-icon-192.png+0−0

Binary file not shown.

public/images/webapp-icon-small.pxm+0−0

Binary file not shown.

public/manifest.json+6−0

   "description": "API Documentation Browser",
   "start_url": "/",
   "display": "standalone",
+  "background_color": "#EEEEEE",
   "icons": [
     {
       "src": "/images/webapp-icon-32.png",
       "sizes": "128x128",
       "type": "image/png"
     },
+    {
+      "src": "/images/webapp-icon-192.png",
+      "sizes": "192x192",
+      "type": "image/png"
+    },
     {
       "src": "/images/webapp-icon-256.png",
       "sizes": "256x256",

test/app_test.rb+5−81

@@ -21,6 +21,11 @@ class AppTest < MiniTest::Spec · 20 unmodified lines
21	assert_equal 'https://example.com/test?q=1', last_response['Location']	21	assert_equal 'https://example.com/test?q=1', last_response['Location']
22	end	22	end
23		23
		24	it 'returns HSTS header' do
		25	get 'https://example.com/test'
		26	assert_equal 'max-age=31536000; includeSubDomains', last_response['Strict-Transport-Security']
		27	end
		28
24	describe "/" do	29	describe "/" do
25	it "works" do	30	it "works" do
26	get '/'	31	get '/'
@@ -38,35 +43,6 @@ class AppTest < MiniTest::Spec · 11 unmodified lines
38	assert last_response.redirect?	43	assert last_response.redirect?
39	assert_equal 'https://example.org/', last_response['Location']	44	assert_equal 'https://example.org/', last_response['Location']
40	end	45	end
41
42	it "sets default size" do
43	get '/'
44	assert_includes last_response.body, 'data-size="20rem"'
45	end
46
47	it "sets size from cookie" do
48	set_cookie('size=42')
49	get '/'
50	assert_includes last_response.body, 'data-size="42px"'
51	end
52
53	it "sets layout from cookie" do
54	set_cookie('layout=foo')
55	get '/'
56	assert_includes last_response.body, '<body class="foo">'
57	end
58
59	it "sets the <html> theme from cookie" do
60	get '/'
61	assert_match %r{<html [^>]class="[^\"]_theme-default}, last_response.body
62	refute_includes last_response.body, '_theme-dark'
63
64	set_cookie('dark=1')
65
66	get '/'
67	assert_match %r{<html [^>]class="[^\"]_theme-dark}, last_response.body
68	refute_includes last_response.body, '_theme-default'
69	end
70	end	46	end
71		47
72	describe "/[static-page]" do	48	describe "/[static-page]" do
@@ -101,58 +77,6 @@ class AppTest < MiniTest::Spec · 28 unmodified lines
101	end	77	end
102	end	78	end
103		79
104	describe "/manifest.appcache" do
105	it "works" do
106	get '/manifest.appcache'
107	assert last_response.ok?
108	end
109
110	it "works with cookie" do
111	set_cookie('docs=css/html~5')
112	get '/manifest.appcache'
113	assert last_response.ok?
114	assert_includes last_response.body, '/css/index.json?1420139788'
115	assert_includes last_response.body, '/html~5/index.json?1420139791'
116	end
117
118	it "ignores invalid docs in the cookie" do
119	set_cookie('docs=foo')
120	get '/manifest.appcache'
121	assert last_response.ok?
122	refute_includes last_response.body, 'foo'
123	end
124
125	it "has the word 'default' when no 'dark' cookie is set" do
126	get '/manifest.appcache'
127	assert_includes last_response.body, '# default'
128	refute_includes last_response.body, '# dark'
129	end
130
131	it "has the word 'dark' when the cookie is set" do
132	set_cookie('dark=1')
133	get '/manifest.appcache'
134	assert_includes last_response.body, '# dark'
135	refute_includes last_response.body, '# default'
136	end
137
138	it "sets default size" do
139	get '/manifest.appcache'
140	assert_includes last_response.body, '20rem'
141	end
142
143	it "sets size from cookie" do
144	set_cookie('size=42')
145	get '/manifest.appcache'
146	assert_includes last_response.body, '42px'
147	end
148
149	it "sets layout from cookie" do
150	set_cookie('layout=foo_layout')
151	get '/manifest.appcache'
152	assert_includes last_response.body, 'foo_layout'
153	end
154	end
155
156	describe "/[doc]" do	80	describe "/[doc]" do
157	it "renders when the doc exists and isn't enabled" do	81	it "renders when the doc exists and isn't enabled" do
158	set_cookie('docs=html~5')	82	set_cookie('docs=html~5')

test/lib/docs/core/scrapers/file_scraper_test.rb+90−48

@@ -2,16 +2,23 @@ require 'test_helper' · 1 unmodified line
2	require 'docs'	2	require 'docs'
3		3
4	class FileScraperTest < MiniTest::Spec	4	class FileScraperTest < MiniTest::Spec
		5	ROOT_PATH = File.expand_path('../../../../../../', __FILE__)
		6
5	class Scraper < Docs::FileScraper	7	class Scraper < Docs::FileScraper
6	self.dir = '/'
7	self.html_filters = Docs::FilterStack.new	8	self.html_filters = Docs::FilterStack.new
8	self.text_filters = Docs::FilterStack.new	9	self.text_filters = Docs::FilterStack.new
		10
		11	version 'version' do; end
9	end	12	end
10		13
11	let :scraper do	14	let :scraper do
12	Scraper.new	15	Scraper.new
13	end	16	end
14		17
		18	let :versioned_scraper do
		19	Scraper.versions.first.new
		20	end
		21
15	let :response do	22	let :response do
16	OpenStruct.new body: 'body', url: Docs::URL.parse(Scraper.base_url)	23	OpenStruct.new body: 'body', url: Docs::URL.parse(Scraper.base_url)
17	end	24	end
@@ -22,9 +29,16 @@ class FileScraperTest < MiniTest::Spec · 4 unmodified lines
22	end	29	end
23	end	30	end
24		31
		32	describe "#source_directory" do
		33	it "returns the directory at docs/[slug]" do
		34	assert_equal File.join(ROOT_PATH, 'docs', 'scraper'), scraper.source_directory
		35	assert_equal File.join(ROOT_PATH, 'docs', 'scraper~version'), versioned_scraper.source_directory
		36	end
		37	end
		38
25	describe "#request_one" do	39	describe "#request_one" do
26	let :path do	40	let :path do
27	File.join(Scraper.dir, 'path')	41	'path'
28	end	42	end
29		43
30	let :result do	44	let :result do
@@ -35,20 +49,34 @@ class FileScraperTest < MiniTest::Spec · 4 unmodified lines
35	stub(scraper).read_file	49	stub(scraper).read_file
36	end	50	end
37		51
38	it "reads a file" do	52	context "when the source directory doesn't exist" do
39	mock(scraper).read_file(path)	53	it "raises an error" do
40	result	54	assert_raises Docs::SetupError do
		55	result
		56	end
		57	end
41	end	58	end
42		59
43	describe "the returned response object" do	60	context "when the source directory exists" do
44	it "has a #body" do	61	before do
45	stub(scraper).read_file { 'body' }	62	stub(scraper).assert_source_directory_exists
46	assert_equal 'body', result.body	63	end
		64
		65	it "reads a file" do
		66	mock(scraper).read_file(File.join(ROOT_PATH, 'docs/scraper', path))
		67	result
47	end	68	end
48		69
49	it "has a #url" do	70	describe "the returned response object" do
50	assert_equal path, result.url.to_s	71	it "has a #body" do
51	assert_instance_of Docs::URL, result.url	72	stub(scraper).read_file { 'body' }
		73	assert_equal 'body', result.body
		74	end
		75
		76	it "has a #url" do
		77	assert_equal path, result.url.to_s
		78	assert_instance_of Docs::URL, result.url
		79	end
52	end	80	end
53	end	81	end
54	end	82	end
@@ -58,49 +86,63 @@ class FileScraperTest < MiniTest::Spec · 3 unmodified lines
58	%w(one two)	86	%w(one two)
59	end	87	end
60		88
61	it "requests the given url" do	89	context "when the source directory doesn't exist" do
62	mock(scraper).request_one('url')	90	it "raises an error" do
63	scraper.send(:request_all, 'url') {}	91	assert_raises Docs::SetupError do
64	end	92	scraper.send(:request_all, urls) {}
65		93	end
66	it "requests the given urls" do	94	end
67	requests = []
68	stub(scraper).request_one { \|url\| requests << url; nil }
69	scraper.send(:request_all, urls) {}
70	assert_equal urls, requests
71	end
72
73	it "yields the responses" do
74	responses = []
75	stub(scraper).request_one { \|url\| urls.index(url) }
76	scraper.send(:request_all, urls) { \|response\| responses << response; nil }
77	assert_equal (0...urls.length).to_a, responses
78	end	95	end
79		96
80	context "when the block returns an array" do	97	context "when the source directory exists" do
81	let :next_urls do	98	before do
82	%w(three four)	99	stub(scraper).assert_source_directory_exists
83	end	100	end
84		101
85	let :all_urls do	102	it "requests the given url" do
86	urls + %w(three four)	103	mock(scraper).request_one('url')
		104	scraper.send(:request_all, 'url') {}
87	end	105	end
88		106
89	it "requests the returned urls" do	107	it "requests the given urls" do
90	requests = []	108	requests = []
91	stub(scraper).request_one { \|url\| requests << url; url }	109	stub(scraper).request_one { \|url\| requests << url; nil }
92	scraper.send(:request_all, urls) { [next_urls.shift].compact }	110	scraper.send(:request_all, urls) {}
93	assert_equal all_urls, requests	111	assert_equal urls, requests
94	end	112	end
95		113
96	it "yields their responses" do	114	it "yields the responses" do
97	responses = []	115	responses = []
98	stub(scraper).request_one { \|url\| all_urls.index(url) }	116	stub(scraper).request_one { \|url\| urls.index(url) }
99	scraper.send :request_all, urls do \|response\|	117	scraper.send(:request_all, urls) { \|response\| responses << response; nil }
100	responses << response	118	assert_equal (0...urls.length).to_a, responses
101	[next_urls.shift].compact	119	end
		120
		121	context "when the block returns an array" do
		122	let :next_urls do
		123	%w(three four)
		124	end
		125
		126	let :all_urls do
		127	urls + %w(three four)
		128	end
		129
		130	it "requests the returned urls" do
		131	requests = []
		132	stub(scraper).request_one { \|url\| requests << url; url }
		133	scraper.send(:request_all, urls) { [next_urls.shift].compact }
		134	assert_equal all_urls, requests
		135	end
		136
		137	it "yields their responses" do
		138	responses = []
		139	stub(scraper).request_one { \|url\| all_urls.index(url) }
		140	scraper.send :request_all, urls do \|response\|
		141	responses << response
		142	[next_urls.shift].compact
		143	end
		144	assert_equal (0...all_urls.length).to_a, responses
102	end	145	end
103	assert_equal (0...all_urls.length).to_a, responses
104	end	146	end
105	end	147	end
106	end	148	end
@@ -123,16 +165,16 @@ class FileScraperTest < MiniTest::Spec · 16 unmodified lines
123		165
124	describe "#read_file" do	166	describe "#read_file" do
125	let :result do	167	let :result do
126	scraper.send :read_file, 'file'	168	scraper.send :read_file, File.join(ROOT_PATH, 'docs', 'scraper', 'file')
127	end	169	end
128		170
129	it "returns the file's content when the file exists" do	171	it "returns the file's content when the file exists in the source directory" do
130	stub(File).read('file') { 'content' }	172	stub(File).read(File.join(ROOT_PATH, 'docs', 'scraper', 'file')) { 'content' }
131	assert_equal 'content', result	173	assert_equal 'content', result
132	end	174	end
133		175
134	it "returns nil when the file doesn't exist" do	176	it "returns nil when the file doesn't exist" do
135	stub(File).read('file') { raise }	177	stub(File).read(File.join(ROOT_PATH, 'docs', 'scraper', 'file')) { raise }
136	assert_nil result	178	assert_nil result
137	end	179	end
138	end	180	end

views/app.erb+1−13

@@ -28,13 +28,7 @@ · 27 unmodified lines
28	</nav>	28	</nav>
29	</header>	29	</header>
30	<section class="_sidebar" tabindex="-1">	30	<section class="_sidebar" tabindex="-1">
31	<div class="_list" role="navigation">	31	<div class="_list" role="navigation"></div>
32	<% unless @doc %>
33	<% App.docs.each do \|slug, doc\| %>
34	<a href="/<%= slug %>/" class="_list-item"><%= doc['full_name'] %></a>
35	<% end %>
36	<% end %>
37	</div>
38	</section>	32	</section>
39	<div class="_container" role="document">	33	<div class="_container" role="document">
40	<main class="_content _content-loading" role="main"></main>	34	<main class="_content _content-loading" role="main"></main>
@@ -63,9 +57,3 @@ · 22 unmodified lines
63	<symbol id="icon-external-link" viewBox="0 0 24 24"><path d="M19 19H5V5h7V3H5c-1.11 0-2 .9-2 2v14c0 1.1.89 2 2 2h14c1.1 0 2-.9 2-2v-7h-2v7zM14 3v2h3.59l-9.83 9.83 1.41 1.41L19 6.41V10h2V3h-7z"/></symbol>	57	<symbol id="icon-external-link" viewBox="0 0 24 24"><path d="M19 19H5V5h7V3H5c-1.11 0-2 .9-2 2v14c0 1.1.89 2 2 2h14c1.1 0 2-.9 2-2v-7h-2v7zM14 3v2h3.59l-9.83 9.83 1.41 1.41L19 6.41V10h2V3h-7z"/></symbol>
64	</defs>	58	</defs>
65	</svg>	59	</svg>
66	<style data-size="<%= app_size %>" data-resizer>
67	._container { margin-left: <%= app_size %>; }
68	._header, ._list { width: <%= app_size %>; }
69	._list-hover.clone { min-width: <%= app_size %>; }
70	._notice, ._path, ._resizer { left: <%= app_size %>; }
71	</style>

views/index.erb+3−2

 <!DOCTYPE html>
-<html<%= ' manifest="/manifest.appcache"' if App.production? %> prefix="og: http://ogp.me/ns#" lang="en" class="_booting _theme-<%= app_theme %>">
+<html prefix="og: http://ogp.me/ns#" lang="en" class="_booting">
 <head>
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no,shrink-to-fit=no">
   <meta name="apple-mobile-web-app-capable" content="yes">
   <meta name="apple-mobile-web-app-status-bar-style" content="black">
   <meta name="format-detection" content="telephone=no">
+  <meta name="theme-color" content="#eee">
   <meta name="robots" content="noodp">
   <title>DevDocs API Documentation</title>
   <link rel="canonical" href="<%= canonical_origin %>">
   <link rel="mask-icon" href="<%= App.cdn_origin %>/images/webkit-mask-icon.svg" color="#398df0">
   <%= stylesheet_tag 'application' %>
 </head>
-<body<%= %( class="#{app_layout}") if app_layout %>>
+<body>
 <noscript class="_fail">DevDocs requires JavaScript to run.</noscript>
 <%= erb :app -%>
 <%= javascript_tag 'application', asset_host: false %>

views/manifest.erb+0−14

@@ -1,14 +0,0 @@
1	CACHE MANIFEST
2	# <%= app_theme %> <%= app_size %> <%= app_layout %>
3
4	CACHE:
5	/
6	<%= manifest_asset_urls.join "\n" %>
7	<%= doc_index_urls.join "\n" %>
8
9	NETWORK:
10	/s/
11	*
12
13	FALLBACK:
14	/ /

views/other.erb+3−2

 <!DOCTYPE html>
-<html lang="en" class="_booting _theme-<%= app_theme %>">
+<html lang="en" class="_booting">
 <head>
   <meta charset="utf-8">
   <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no">
   <% if doc_index_page? %><meta name="description" content="<%= @doc['name'] %> <%= @doc['release'] %> API documentation with instant search, offline support, keyboard shortcuts, mobile version, and more."><% else %><meta name="robots" content="noindex"><% end %>
   <meta name="format-detection" content="telephone=no">
+  <meta name="theme-color" content="#eee">
   <meta property="og:image" content="<%= App.cdn_origin %>/images/icon-320.png">
   <title>DevDocs<%= " &mdash; #{@doc['full_name']} documentation" if doc_index_page? %></title>
   <link rel="canonical" href="<%= canonical_origin %><%= request.path %>">
   <link rel="search" type="application/opensearchdescription+xml" href="/opensearch.xml" title="Search DevDocs">
   <%= stylesheet_tag 'application' %>
 </head>
-<body<%= %( class="#{app_layout}") if app_layout %> data-doc="<%= CGI::escape_html @doc.to_json %>">
+<body data-doc="<%= CGI::escape_html @doc.to_json %>">
 <noscript class="_fail">DevDocs requires JavaScript to run.</noscript>
 <%= erb :app -%>
 <%= javascript_tag 'application', asset_host: false %><% unless App.production? %>

views/service-worker.js.erb+61−0

@@ -0,0 +1,61 @@
	1	<%# The name of the cache to store responses in %>
	2	<%# If the cache name changes DevDocs is assumed to be updated %>
	3	const cacheName = '<%= service_worker_cache_name %>';
	4
	5	<%# Url's to cache when the service worker is installed %>
	6	const urlsToCache = [
	7	'/',
	8	'/favicon.ico',
	9	'/manifest.json',
	10	'<%= service_worker_asset_urls.join "',\n '" %>',
	11	'<%= doc_index_urls.join "',\n '" %>',
	12	];
	13
	14	<%# Set-up the cache %>
	15	self.addEventListener('install', event => {
	16	self.skipWaiting();
	17
	18	event.waitUntil(
	19	caches.open(cacheName).then(cache => cache.addAll(urlsToCache)),
	20	);
	21	});
	22
	23	<%# Remove old caches %>
	24	self.addEventListener('activate', event => {
	25	event.waitUntil((async () => {
	26	const keys = await caches.keys();
	27	const jobs = keys.map(key => key !== cacheName ? caches.delete(key) : Promise.resolve());
	28	return Promise.all(jobs);
	29	})());
	30	});
	31
	32	<%# Handle HTTP requests %>
	33	self.addEventListener('fetch', event => {
	34	event.respondWith((async () => {
	35	const cachedResponse = await caches.match(event.request);
	36	if (cachedResponse) return cachedResponse;
	37
	38	try {
	39	const response = await fetch(event.request);
	40
	41	if (!response.ok) {
	42	throw new Error(`The HTTP request failed with status code ${response.status}`);
	43	}
	44
	45	return response;
	46	} catch (err) {
	47	const url = new URL(event.request.url);
	48
	49	<%# Attempt to return the index page from the cache if the user is visiting a url like devdocs.io/offline or devdocs.io/javascript/global_objects/array/find %>
	50	<%# The index page will make sure the correct documentation or a proper offline page is shown %>
	51	const pathname = url.pathname;
	52	const filename = pathname.substr(1 + pathname.lastIndexOf('/')).split(/\#\|\?/g)[0];
	53	if (url.origin === location.origin && !filename.includes('.')) {
	54	const cachedIndex = await caches.match('/');
	55	if (cachedIndex) return cachedIndex;
	56	}
	57
	58	throw err;
	59	}
	60	})());
	61	});

views/unsupported.erb+6−6

@@ -11,16 +11,16 @@ · 10 unmodified lines
11	<p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:</p>	11	<p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:</p>
12	<ul class="_fail-list">	12	<ul class="_fail-list">
13	<li>Recent versions of Firefox, Chrome, or Opera</li>	13	<li>Recent versions of Firefox, Chrome, or Opera</li>
14	<li>Safari 9.1+</li>	14	<li>Safari 11.1+</li>
15	<li>Edge 16+</li>	15	<li>Edge 17+</li>
16	<li>iOS 10+</li>	16	<li>iOS 11.3+</li>
17	</ul>	17	</ul>
18	<p class="_fail-text">	18	<p class="_fail-text">
19	If you're unable to upgrade, I apologize.	19	If you're unable to upgrade, we apologize.
20	I decided to prioritize speed and new features over support for older browsers.	20	We decided to prioritize speed and new features over support for older browsers.
21	</p>	21	</p>
22	<p class="_fail-text">	22	<p class="_fail-text">
23	— Thibaut <a href="https://twitter.com/DevDocs" class="_fail-link">@DevDocs</a>	23	— <a href="https://twitter.com/DevDocs">@DevDocs</a>
24	</p>	24	</p>
25	</div>	25	</div>
26	</body>	26	</body>

@@ -8,13 +8,13 @@ app.templates.intro = """ · 7 unmodified lines
8	<p>Thanks for downloading DevDocs. Here are a few things you should know:	8	<p>Thanks for downloading DevDocs. Here are a few things you should know:
9	<ol class="_intro-list">	9	<ol class="_intro-list">
10	<li>Your local version of DevDocs won't self-update. Unless you're modifying the code,	10	<li>Your local version of DevDocs won't self-update. Unless you're modifying the code,
11	I recommend using the hosted version at <a href="https://devdocs.io">devdocs.io</a>.	11	we recommend using the hosted version at <a href="https://devdocs.io">devdocs.io</a>.
12	<li>Run <code>thor docs:list</code> to see all available documentations.	12	<li>Run <code>thor docs:list</code> to see all available documentations.
13	<li>Run <code>thor docs:download <name></code> to download documentations.	13	<li>Run <code>thor docs:download <name></code> to download documentations.
14	<li>Run <code>thor docs:download --installed</code> to update all downloaded documentations.	14	<li>Run <code>thor docs:download --installed</code> to update all downloaded documentations.
15	<li>To be notified about new versions, don't forget to <a href="https://github.com/freeCodeCamp/devdocs/subscription">watch the repository</a> on GitHub.	15	<li>To be notified about new versions, don't forget to <a href="https://github.com/freeCodeCamp/devdocs/subscription">watch the repository</a> on GitHub.
16	<li>The <a href="https://github.com/freeCodeCamp/devdocs/issues">issue tracker</a> is the preferred channel for bug reports and	16	<li>The <a href="https://github.com/freeCodeCamp/devdocs/issues">issue tracker</a> is the preferred channel for bug reports and
17	feature requests. For everything else, use the <a href="https://groups.google.com/d/forum/devdocs">mailing list</a>.	17	feature requests. For everything else, use <a href="https://gitter.im/FreeCodeCamp/DevDocs">Gitter</a>.
18	<li>Contributions are welcome. See the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/CONTRIBUTING.md">guidelines</a>.	18	<li>Contributions are welcome. See the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/CONTRIBUTING.md">guidelines</a>.
19	<li>DevDocs is licensed under the terms of the Mozilla Public License v2.0. For more information,	19	<li>DevDocs is licensed under the terms of the Mozilla Public License v2.0. For more information,
20	see the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/COPYRIGHT">COPYRIGHT</a> and	20	see the <a href="https://github.com/freeCodeCamp/devdocs/blob/master/COPYRIGHT">COPYRIGHT</a> and
@@ -39,7 +39,7 @@ app.templates.intro = """ · 18 unmodified lines
39	<li>DevDocs works <a href="/offline">offline</a>, on mobile, and can be installed on <a href="https://chrome.google.com/webstore/detail/devdocs/mnfehgbmkapmjnhcnbodoamcioleeooe">Chrome</a>.	39	<li>DevDocs works <a href="/offline">offline</a>, on mobile, and can be installed on <a href="https://chrome.google.com/webstore/detail/devdocs/mnfehgbmkapmjnhcnbodoamcioleeooe">Chrome</a>.
40	<li>For the latest news, follow <a href="https://twitter.com/DevDocs">@DevDocs</a>.	40	<li>For the latest news, follow <a href="https://twitter.com/DevDocs">@DevDocs</a>.
41	<li>DevDocs is free and <a href="https://github.com/freeCodeCamp/devdocs">open source</a>.	41	<li>DevDocs is free and <a href="https://github.com/freeCodeCamp/devdocs">open source</a>.
42	<iframe class="_github-btn" src="//ghbtns.com/github-btn.html?user=freeCodeCamp&repo=devdocs&type=watch&count=true" allowtransparency="true" frameborder="0" scrolling="0" width="100" height="20"></iframe>	42	<object data="https://img.shields.io/github/stars/freeCodeCamp/devdocs.svg?style=social" type="image/svg+xml" aria-hidden="true"></object>
43	<li>And if you're new to coding, check out <a href="https://www.freecodecamp.org/">freeCodeCamp's open source curriculum</a>.	43	<li>And if you're new to coding, check out <a href="https://www.freecodecamp.org/">freeCodeCamp's open source curriculum</a>.
44	</ol>	44	</ol>
45	<p>Happy coding!	45	<p>Happy coding!