History

Fix merge conflicts

Jasper van Merle committed 7 years ago

Commit 54381432 parents 4355631 ac50e9943 files · +760 −254Browse files

Unified · Split

43 files
  1. .editorconfig +3 −0
  2. .github/CONTRIBUTING.md +3 −3
  3. .gitignore +1 −0
  4. .travis.yml +8 −0
  5. CODE_OF_CONDUCT.md +2 −0
  6. Dockerfile +1 −0
  7. Dockerfile-alpine +1 −0
  8. PULL_REQUEST_TEMPLATE.md +11 −0
  9. README.md +36 −10
  10. assets/javascripts/app/app.coffee +9 −7
  11. assets/javascripts/app/appcache.coffee +0 −42
  12. assets/javascripts/app/config.coffee.erb +2 −0
  13. assets/javascripts/app/serviceworker.coffee +49 −0
  14. assets/javascripts/app/settings.coffee +30 −0
  15. assets/javascripts/app/update_checker.coffee +3 −3
  16. assets/javascripts/lib/util.coffee +4 −0
  17. assets/javascripts/news.json +4 −0
  18. assets/javascripts/templates/error_tmpl.coffee +5 −5
  19. assets/javascripts/templates/pages/offline_tmpl.coffee +9 −4
  20. assets/javascripts/templates/pages/root_tmpl.coffee.erb +2 −2
  21. assets/javascripts/views/content/entry_page.coffee +1 −1
  22. assets/javascripts/views/content/settings_page.coffee +1 −11
  23. assets/javascripts/views/layout/resizer.coffee +2 −9
  24. assets/javascripts/views/layout/settings.coffee +1 −9
  25. assets/javascripts/views/search/search.coffee +8 −0
  26. assets/javascripts/views/search/search_scope.coffee +23 −5
  27. assets/javascripts/views/sidebar/sidebar.coffee +1 −1
  28. assets/stylesheets/components/_header.scss +1 −0
  29. docs/adding-docs.md +23 −0
  30. docs/filter-reference.md +224 −0
  31. docs/maintainers.md +7 −7
  32. docs/scraper-reference.md +186 −0
  33. lib/app.rb +35 −20
  34. public/docs/docs.json +0 −1
  35. public/images/webapp-icon-192.png +0 −0
  36. public/manifest.json +6 −0
  37. test/app_test.rb +0 −81
  38. views/app.erb +1 −13
  39. views/index.erb +3 −2
  40. views/manifest.erb +0 −14
  41. views/other.erb +3 −2
  42. views/service-worker.js.erb +49 −0
  43. views/unsupported.erb +3 −3
.editorconfig+3−0
@@ -6,3 +6,6 @@ indent_style = space · 5 unmodified lines
6indent_size = 26indent_size = 2
7insert_final_newline = true7insert_final_newline = true
8trim_trailing_whitespace = true8trim_trailing_whitespace = true
9
10[*.md]
11trim_trailing_whitespace = false
.github/CONTRIBUTING.md+3−3
@@ -24,12 +24,12 @@ Want to contribute? Great. Please review the following guidelines carefully and · 23 unmodified lines
24## Requesting new features24## Requesting new features
25 25
261. Search for similar feature requests; someone may have already requested it.261. Search for similar feature requests; someone may have already requested it.
272. Make sure your feature fits DevDocs's [vision](https://github.com/freeCodeCamp/devdocs/blob/master/README.md#vision).272. Make sure your feature fits DevDocs's [vision](../README.md#vision).
283. Provide a clear and detailed explanation of the feature and why it's important to add it.283. Provide a clear and detailed explanation of the feature and why it's important to add it.
29 29
30## Requesting new documentations30## Requesting new documentations
31 31
32Please don't open issues to request new documentations. 32Please don't open issues to request new documentations.
33Use the [Trello board](https://trello.com/b/6BmTulfx/devdocs-documentation) where everyone can vote.33Use the [Trello board](https://trello.com/b/6BmTulfx/devdocs-documentation) where everyone can vote.
34 34
35## Contributing code and features35## Contributing code and features
@@ -44,7 +44,7 @@ Use the [Trello board](https://trello.com/b/6BmTulfx/devdocs-documentation) wher · 8 unmodified lines
44 44
45## Contributing new documentations45## Contributing new documentations
46 46
47See the [wiki](https://github.com/freeCodeCamp/devdocs/wiki) to learn how to add new documentations.47See the [`docs` folder](https://github.com/freeCodeCamp/devdocs/tree/master/docs) to learn how to add new documentations.
48 48
49**Important:** the documentation's license must permit alteration, redistribution and commercial use, and the documented software must be released under an open source license. Feel free to get in touch if you're not sure if a documentation meets those requirements.49**Important:** the documentation's license must permit alteration, redistribution and commercial use, and the documented software must be released under an open source license. Feel free to get in touch if you're not sure if a documentation meets those requirements.
50 50
.gitignore+1−0
@@ -1,5 +1,6 @@
1.DS_Store1.DS_Store
2.bundle2.bundle
3log
3tmp4tmp
4public/assets5public/assets
5public/fonts6public/fonts
.travis.yml+8−0
@@ -5,3 +5,11 @@ cache: bundler · 4 unmodified lines
5before_script:5before_script:
6 - gem update --system6 - gem update --system
7 - gem install bundler7 - gem install bundler
8
9deploy:
10 provider: heroku
11 app: devdocs
12 on:
13 branch: master
14 api_key:
15 secure: "4p1klvWJZSOImzFcKOduILjP93hlOlAhceWlYMKS4tU+TCFE8qTBzdKdFPSCsCgjB+YR9pBss+L0lJpVVMjSwFHXqpKe6EeUSltO2k7DFHfW7kXLUM/L0AfqXz+YXk76XUyZMhvOEbldPfaMaj10e8vgDOQCSHABDyK/4CU+hnI="
CODE_OF_CONDUCT.md+2−0
@@ -0,0 +1,2 @@
1
2> Our Code of Conduct is available here: <https://code-of-conduct.freecodecamp.org/>
Dockerfile+1−0
@@ -1,6 +1,7 @@
1FROM ruby:2.6.01FROM ruby:2.6.0
2 2
3ENV LANG=C.UTF-83ENV LANG=C.UTF-8
4ENV ENABLE_SERVICE_WORKER=true
4 5
5WORKDIR /devdocs6WORKDIR /devdocs
6 7
Dockerfile-alpine+1−0
@@ -1,6 +1,7 @@
1FROM ruby:2.6.0-alpine1FROM ruby:2.6.0-alpine
2 2
3ENV LANG=C.UTF-83ENV LANG=C.UTF-8
4ENV ENABLE_SERVICE_WORKER=true
4 5
5WORKDIR /devdocs6WORKDIR /devdocs
6 7
PULL_REQUEST_TEMPLATE.md+11−0
@@ -0,0 +1,11 @@
1If you’re adding a new scraper, please ensure that you have:
2
3- [ ] Tested the scraper on a local copy of DevDocs
4- [ ] Ensured that the docs are styled similarly to other docs on DevDocs
5<!-- If the docs don’t have an icon, delete the next four items: -->
6- [ ] Added these files to the <code>public/icons/*your_scraper_name*/</code> directory:
7 - [ ] `16.png`: a 16×16 pixel icon for the doc
8 - [ ] `16@2x.png`: a 32×32 pixel icon for the doc
9 - [ ] `SOURCE`: A text file containing the URL to the page the image can be found on or the URL of the original image itself
10
11<!-- Replace the `[ ]` with a `[x]` once you’ve completed each step. -->
README.md+36−10
@@ -10,7 +10,7 @@ Keep track of development news: · 9 unmodified lines
10* Watch the repository on [GitHub](https://github.com/freeCodeCamp/devdocs/subscription)10* Watch the repository on [GitHub](https://github.com/freeCodeCamp/devdocs/subscription)
11* Follow [@DevDocs](https://twitter.com/DevDocs) on Twitter11* Follow [@DevDocs](https://twitter.com/DevDocs) on Twitter
12 12
13**Table of Contents:** [Quick Start](#quick-start) · [Vision](#vision) · [App](#app) · [Scraper](#scraper) · [Commands](#available-commands) · [Contributing](#contributing) · [License](#copyright--license) · [Questions?](#questions)13**Table of Contents:** [Quick Start](#quick-start) · [Vision](#vision) · [App](#app) · [Scraper](#scraper) · [Commands](#available-commands) · [Contributing](#contributing) · [Documentation](#documentation) · [Plugins and Extensions](#plugins-and-extensions) · [License](#copyright--license) · [Questions?](#questions)
14 14
15## Quick Start15## Quick Start
16 16
@@ -59,14 +59,14 @@ The web app is all client-side JavaScript, written in [CoffeeScript](http://coff · 42 unmodified lines
59 59
60Many of the code's design decisions were driven by the fact that the app uses XHR to load content directly into the main frame. This includes stripping the original documents of most of their HTML markup (e.g. scripts and stylesheets) to avoid polluting the main frame, and prefixing all CSS class names with an underscore to prevent conflicts.60Many of the code's design decisions were driven by the fact that the app uses XHR to load content directly into the main frame. This includes stripping the original documents of most of their HTML markup (e.g. scripts and stylesheets) to avoid polluting the main frame, and prefixing all CSS class names with an underscore to prevent conflicts.
61 61
62Another driving factor is performance and the fact that everything happens in the browser. `applicationCache` (which comes with its own set of constraints) and `localStorage` are used to speed up the boot time, while memory consumption is kept in check by allowing the user to pick his/her own set of documentations. The search algorithm is kept simple because it needs to be fast even searching through 100,000 strings.62Another driving factor is performance and the fact that everything happens in the browser. A service worker (which comes with its own set of constraints) and `localStorage` are used to speed up the boot time, while memory consumption is kept in check by allowing the user to pick his/her own set of documentations. The search algorithm is kept simple because it needs to be fast even searching through 100,000 strings.
63 63
64DevDocs being a developer tool, the browser requirements are high:64DevDocs being a developer tool, the browser requirements are high:
65 65
66* Recent versions of Firefox, Chrome, or Opera66* Recent versions of Firefox, Chrome, or Opera
67* Safari 9.1+67* Safari 11.1+
68* Edge 16+68* Edge 17+
69* iOS 10+69* iOS 11.3+
70 70
71This allows the code to take advantage of the latest DOM and HTML5 APIs and make developing DevDocs a lot more fun!71This allows the code to take advantage of the latest DOM and HTML5 APIs and make developing DevDocs a lot more fun!
72 72
@@ -83,12 +83,13 @@ Modifications made to each document include: · 10 unmodified lines
83* replacing all external (not scraped) URLs with their fully qualified counterpart83* replacing all external (not scraped) URLs with their fully qualified counterpart
84* replacing all internal (scraped) URLs with their unqualified and relative counterpart84* replacing all internal (scraped) URLs with their unqualified and relative counterpart
85* adding content, such as a title and link to the original document85* adding content, such as a title and link to the original document
86* ensuring correct syntax highlighting using [Prism](http://prismjs.com/)
86 87
87These modifications are applied via a set of filters using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. Each scraper includes filters specific to itself, one of which is tasked with figuring out the pages' metadata.88These modifications are applied via a set of filters using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. Each scraper includes filters specific to itself, one of which is tasked with figuring out the pages' metadata.
88 89
89The end result is a set of normalized HTML partials and two JSON files (index + offline data). Because the index files are loaded separately by the [app](#app) following the user's preferences, the scraper also creates a JSON manifest file containing information about the documentations currently available on the system (such as their name, version, update date, etc.).90The end result is a set of normalized HTML partials and two JSON files (index + offline data). Because the index files are loaded separately by the [app](#app) following the user's preferences, the scraper also creates a JSON manifest file containing information about the documentations currently available on the system (such as their name, version, update date, etc.).
90 91
91More information about scrapers and filters is available on the [wiki](https://github.com/freeCodeCamp/devdocs/wiki).92More information about [scrapers](./docs/scraper-reference.md) and [filters](./docs/filter-reference.md) is available in the `docs` folder.
92 93
93## Available Commands94## Available Commands
94 95
@@ -128,15 +129,40 @@ If multiple versions of Ruby are installed on your system, commands must be run · 33 unmodified lines
128 129
129## Contributing130## Contributing
130 131
131Contributions are welcome. Please read the [contributing guidelines](https://github.com/freeCodeCamp/devdocs/blob/master/.github/CONTRIBUTING.md).132Contributions are welcome. Please read the [contributing guidelines](./.github/CONTRIBUTING.md).
132 133
133DevDocs's own documentation is available on the [wiki](https://github.com/freeCodeCamp/devdocs/wiki).134## Documentation
135
136* [Adding documentations to DevDocs](./docs/adding-docs.md)
137* [Scraper Reference](./docs/scraper-reference.md)
138* [Filter Reference](./docs/filter-reference.md)
139* [Maintainers’ Guide](./docs/maintainers.md)
140
141## Plugins and Extensions
142
143* [Chrome web app](https://chrome.google.com/webstore/detail/devdocs/mnfehgbmkapmjnhcnbodoamcioleeooe)
144* [Ubuntu Touch app](https://uappexplorer.com/app/devdocsunofficial.berkes)
145* [Sublime Text plugin](https://sublime.wbond.net/packages/DevDocs)
146* [Atom plugin](https://atom.io/packages/devdocs)
147* [Brackets extension](https://github.com/gruehle/dev-docs-viewer)
148* [Fluid](http://fluidapp.com) for turning DevDocs into a real OS X app
149* [GTK shell / Vim integration](https://github.com/naquad/devdocs-shell)
150* [Emacs lookup](https://github.com/skeeto/devdocs-lookup)
151* [Alfred Workflow](https://github.com/yannickglt/alfred-devdocs)
152* [Vim search plugin with Devdocs in its defaults](https://github.com/waiting-for-dev/vim-www) Just set `let g:www_shortcut_engines = { 'devdocs': ['Devdocs', '<leader>dd'] }` to have a `:Devdocs` command and a `<leader>dd` mapping.
153* [Visual Studio Code plugin](https://marketplace.visualstudio.com/items?itemName=akfish.vscode-devdocs ) (1)
154* [Visual Studio Code plugin](https://marketplace.visualstudio.com/items?itemName=deibit.devdocs) (2)
155* [Desktop application](https://github.com/egoist/devdocs-desktop)
156* [Doc Browser](https://github.com/qwfy/doc-browser) is a native Linux app that supports DevDocs docsets
157* [GNOME Application](https://github.com/hardpixel/devdocs-desktop) GTK3 application with search integrated in headerbar
158* [macOS Application](https://github.com/dteoh/devdocs-macos)
159* [Android Application](https://github.com/Merith-TK/devdocs_webapp_kotlin) is a fully working, advanced WebView
134 160
135## Copyright / License161## Copyright / License
136 162
137Copyright 2013-2019 Thibaut Courouble and [other contributors](https://github.com/freeCodeCamp/devdocs/graphs/contributors)163Copyright 2013-2019 Thibaut Courouble and [other contributors](https://github.com/freeCodeCamp/devdocs/graphs/contributors)
138 164
139This software is licensed under the terms of the Mozilla Public License v2.0. See the [COPYRIGHT](https://github.com/freeCodeCamp/devdocs/blob/master/COPYRIGHT) and [LICENSE](https://github.com/freeCodeCamp/devdocs/blob/master/LICENSE) files.165This software is licensed under the terms of the Mozilla Public License v2.0. See the [COPYRIGHT](./COPYRIGHT) and [LICENSE](./LICENSE) files.
140 166
141Please do not use the name DevDocs to endorse or promote products derived from this software without the maintainers' permission, except as may be necessary to comply with the notice/attribution requirements.167Please do not use the name DevDocs to endorse or promote products derived from this software without the maintainers' permission, except as may be necessary to comply with the notice/attribution requirements.
142 168
assets/javascripts/app/app.coffee+9−7
@@ -13,10 +13,12 @@ · 12 unmodified lines
13 13
14 @el = $('._app')14 @el = $('._app')
15 @localStorage = new LocalStorageStore15 @localStorage = new LocalStorageStore
16 @appCache = new app.AppCache if app.AppCache.isEnabled()16 @serviceWorker = new app.ServiceWorker if app.ServiceWorker.isEnabled()
17 @settings = new app.Settings17 @settings = new app.Settings
18 @db = new app.DB()18 @db = new app.DB()
19 19
20 @settings.initLayout()
21
20 @docs = new app.collections.Docs22 @docs = new app.collections.Docs
21 @disabledDocs = new app.collections.Docs23 @disabledDocs = new app.collections.Docs
22 @entries = new app.collections.Entries24 @entries = new app.collections.Entries
@@ -147,7 +149,7 @@ · 124 unmodified lines
147 saveDocs: ->149 saveDocs: ->
148 @settings.setDocs(doc.slug for doc in @docs.all())150 @settings.setDocs(doc.slug for doc in @docs.all())
149 @db.migrate()151 @db.migrate()
150 @appCache?.updateInBackground()152 @serviceWorker?.updateInBackground()
151 153
152 welcomeBack: ->154 welcomeBack: ->
153 visitCount = @settings.get('count')155 visitCount = @settings.get('count')
@@ -167,14 +169,14 @@ · 13 unmodified lines
167 reload: ->169 reload: ->
168 @docs.clearCache()170 @docs.clearCache()
169 @disabledDocs.clearCache()171 @disabledDocs.clearCache()
170 if @appCache then @appCache.reload() else @reboot()172 if @serviceWorker then @serviceWorker.reload() else @reboot()
171 return173 return
172 174
173 reset: ->175 reset: ->
174 @localStorage.reset()176 @localStorage.reset()
175 @settings.reset()177 @settings.reset()
176 @db?.reset()178 @db?.reset()
177 @appCache?.update()179 @serviceWorker?.update()
178 window.location = '/'180 window.location = '/'
179 return181 return
180 182
@@ -193,9 +195,9 @@ · 12 unmodified lines
193 return195 return
194 196
195 indexHost: ->197 indexHost: ->
196 # Can't load the index files from the host/CDN when applicationCache is198 # Can't load the index files from the host/CDN when service worker is
197 # enabled because it doesn't support caching URLs that use CORS.199 # enabled because it doesn't support caching URLs that use CORS.
198 @config[if @appCache and @settings.hasDocs() then 'index_path' else 'docs_origin']200 @config[if @serviceWorker and @settings.hasDocs() then 'index_path' else 'docs_origin']
199 201
200 onBootError: (args...) ->202 onBootError: (args...) ->
201 @trigger 'bootError'203 @trigger 'bootError'
@@ -252,7 +254,7 @@ · 50 unmodified lines
252 matchMedia: !!window.matchMedia254 matchMedia: !!window.matchMedia
253 insertAdjacentHTML: !!document.body.insertAdjacentHTML255 insertAdjacentHTML: !!document.body.insertAdjacentHTML
254 defaultPrevented: document.createEvent('CustomEvent').defaultPrevented is false256 defaultPrevented: document.createEvent('CustomEvent').defaultPrevented is false
255 cssVariables: CSS.supports and CSS.supports('(--t: 0)')257 cssVariables: !!CSS?.supports?('(--t: 0)')
256 258
257 for key, value of features when not value259 for key, value of features when not value
258 Raven.captureMessage "unsupported/#{key}", level: 'info'260 Raven.captureMessage "unsupported/#{key}", level: 'info'
assets/javascripts/app/appcache.coffee+0−42
@@ -1,42 +0,0 @@
1class app.AppCache
2 $.extend @prototype, Events
3
4 @isEnabled: ->
5 try
6 applicationCache and applicationCache.status isnt applicationCache.UNCACHED
7 catch
8
9 constructor: ->
10 @cache = applicationCache
11 @notifyUpdate = true
12 @onUpdateReady() if @cache.status is @cache.UPDATEREADY
13
14 $.on @cache, 'progress', @onProgress
15 $.on @cache, 'updateready', @onUpdateReady
16
17 update: ->
18 @notifyUpdate = true
19 @notifyProgress = true
20 try @cache.update() catch
21 return
22
23 updateInBackground: ->
24 @notifyUpdate = false
25 @notifyProgress = false
26 try @cache.update() catch
27 return
28
29 reload: ->
30 $.on @cache, 'updateready noupdate error', -> app.reboot()
31 @notifyUpdate = false
32 @notifyProgress = true
33 try @cache.update() catch
34 return
35
36 onProgress: (event) =>
37 @trigger 'progress', event if @notifyProgress
38 return
39
40 onUpdateReady: =>
41 @trigger 'updateready' if @notifyUpdate
42 return
assets/javascripts/app/config.coffee.erb+2−0
@@ -13,3 +13,5 @@ app.config = · 12 unmodified lines
13 version: <%= Time.now.to_i %>13 version: <%= Time.now.to_i %>
14 release: <%= Time.now.utc.httpdate.to_json %>14 release: <%= Time.now.utc.httpdate.to_json %>
15 mathml_stylesheet: '<%= App.cdn_origin %>/mathml.css'15 mathml_stylesheet: '<%= App.cdn_origin %>/mathml.css'
16 service_worker_path: '/service-worker.js'
17 service_worker_enabled: <%= App.environment == :production || ENV['ENABLE_SERVICE_WORKER'] == 'true' %>
assets/javascripts/app/serviceworker.coffee+49−0
@@ -0,0 +1,49 @@
1class app.ServiceWorker
2 $.extend @prototype, Events
3
4 @isEnabled: ->
5 !!navigator.serviceWorker and app.config.service_worker_enabled
6
7 constructor: ->
8 @registration = null
9 @notifyUpdate = true
10
11 navigator.serviceWorker.register(app.config.service_worker_path, {scope: '/'})
12 .then(
13 (registration) => @updateRegistration(registration),
14 (error) -> console.error('Could not register service worker:', error)
15 )
16
17 update: ->
18 return unless @registration
19 @notifyUpdate = true
20 return @registration.update().catch(->)
21
22 updateInBackground: ->
23 return unless @registration
24 @notifyUpdate = false
25 return @registration.update().catch(->)
26
27 reload: ->
28 return @updateInBackground().then(() -> app.reboot())
29
30 updateRegistration: (registration) ->
31 @registration = registration
32 $.on @registration, 'updatefound', @onUpdateFound
33 return
34
35 onUpdateFound: =>
36 $.off @installingRegistration, 'statechange', @onStateChange() if @installingRegistration
37 @installingRegistration = @registration.installing
38 $.on @installingRegistration, 'statechange', @onStateChange
39 return
40
41 onStateChange: =>
42 if @installingRegistration and @installingRegistration.state == 'installed' and navigator.serviceWorker.controller
43 @installingRegistration = null
44 @onUpdateReady()
45 return
46
47 onUpdateReady: ->
48 @trigger 'updateready' if @notifyUpdate
49 return
assets/javascripts/app/settings.coffee+30−0
@@ -20,6 +20,8 @@ class app.Settings · 19 unmodified lines
20 'news'20 'news'
21 ]21 ]
22 22
23 LAYOUTS: ['_max-width', '_sidebar-hidden', '_native-scrollbars']
24
23 @defaults:25 @defaults:
24 count: 026 count: 0
25 hideDisabled: false27 hideDisabled: false
@@ -40,6 +42,7 @@ class app.Settings · 14 unmodified lines
40 set: (key, value) ->42 set: (key, value) ->
41 @store.set(key, value)43 @store.set(key, value)
42 delete @cache[key]44 delete @cache[key]
45 @toggleDark(value) if key == 'dark'
43 return46 return
44 47
45 del: (key) ->48 del: (key) ->
@@ -65,6 +68,8 @@ class app.Settings · 19 unmodified lines
65 return68 return
66 69
67 setLayout: (name, enable) ->70 setLayout: (name, enable) ->
71 @toggleLayout(name, enable)
72
68 layout = (@store.get('layout') || '').split(' ')73 layout = (@store.get('layout') || '').split(' ')
69 $.arrayDelete(layout, '')74 $.arrayDelete(layout, '')
70 75
@@ -106,3 +111,28 @@ class app.Settings · 35 unmodified lines
106 @store.reset()111 @store.reset()
107 @cache = {}112 @cache = {}
108 return113 return
114
115 initLayout: ->
116 @toggleDark(@get('dark') is 1)
117 @toggleLayout(layout, @hasLayout(layout)) for layout in @LAYOUTS
118 @initSidebarWidth()
119 return
120
121 toggleDark: (enable) ->
122 classList = document.documentElement.classList
123 classList.toggle('_theme-default', !enable)
124 classList.toggle('_theme-dark', enable)
125 color = getComputedStyle(document.documentElement).getPropertyValue('--headerBackground').trim()
126 $('meta[name=theme-color]').setAttribute('content', color)
127 return
128
129 toggleLayout: (layout, enable) ->
130 classList = document.body.classList
131 classList.toggle(layout, enable) unless layout is '_sidebar-hidden'
132 classList.toggle('_overlay-scrollbars', $.overlayScrollbarsEnabled())
133 return
134
135 initSidebarWidth: ->
136 size = @get('size')
137 document.documentElement.style.setProperty('--sidebarWidth', size + 'px') if size
138 return
assets/javascripts/app/update_checker.coffee+3−3
@@ -3,13 +3,13 @@ class app.UpdateChecker · 2 unmodified lines
3 @lastCheck = Date.now()3 @lastCheck = Date.now()
4 4
5 $.on window, 'focus', @onFocus5 $.on window, 'focus', @onFocus
6 app.appCache.on 'updateready', @onUpdateReady if app.appCache6 app.serviceWorker?.on 'updateready', @onUpdateReady
7 7
8 setTimeout @checkDocs, 08 setTimeout @checkDocs, 0
9 9
10 check: ->10 check: ->
11 if app.appCache11 if app.serviceWorker
12 app.appCache.update()12 app.serviceWorker.update()
13 else13 else
14 ajax14 ajax
15 url: $('script[src*="application"]').getAttribute('src')15 url: $('script[src*="application"]').getAttribute('src')
assets/javascripts/lib/util.coffee+4−0
@@ -352,6 +352,10 @@ isIE = null · 351 unmodified lines
352$.isIE = ->352$.isIE = ->
353 isIE ?= navigator.userAgent?.indexOf('MSIE') >= 0 || navigator.userAgent?.indexOf('rv:11.0') >= 0353 isIE ?= navigator.userAgent?.indexOf('MSIE') >= 0 || navigator.userAgent?.indexOf('rv:11.0') >= 0
354 354
355isChromeForAndroid = null
356$.isChromeForAndroid = ->
357 isChromeForAndroid ?= navigator.userAgent?.indexOf('Android') >= 0 && /Chrome\/([.0-9])+ Mobile/.test(navigator.userAgent)
358
355isAndroid = null359isAndroid = null
356$.isAndroid = ->360$.isAndroid = ->
357 isAndroid ?= navigator.userAgent?.indexOf('Android') >= 0361 isAndroid ?= navigator.userAgent?.indexOf('Android') >= 0
assets/javascripts/news.json+4−0
@@ -1,4 +1,8 @@
1[1[
2 [
3 "2019-07-19",
4 "Replaced the AppCache with a Service Worker (which makes DevDocs an installable PWA) and fixed layout preferences on Firefox."
5 ],
2 [6 [
3 "2018-09-23",7 "2018-09-23",
4 "New documentations: <a href=\"/puppeteer/\">Puppeteer</a> and <a href=\"/handlebars/\">Handlebars.js</a>"8 "New documentations: <a href=\"/puppeteer/\">Puppeteer</a> and <a href=\"/handlebars/\">Handlebars.js</a>"
assets/javascripts/templates/error_tmpl.coffee+5−5
@@ -12,8 +12,8 @@ app.templates.notFoundPage = -> · 11 unmodified lines
12 12
13app.templates.pageLoadError = ->13app.templates.pageLoadError = ->
14 error """ The page failed to load. """,14 error """ The page failed to load. """,
15 """ It may be missing from the server (try reloading the app) or you could be offline.<br>15 """ It may be missing from the server (try reloading the app) or you could be offline (try <a href="/offline">installing the documentation for offline usage</a> when online again).<br>
16 If you keep seeing this, you're likely behind a proxy or firewall that blocks cross-domain requests. """,16 If you're online and you keep seeing this, you're likely behind a proxy or firewall that blocks cross-domain requests. """,
17 """ #{back} &middot; <a href="/##{location.pathname}" target="_top" class="_error-link">Reload</a>17 """ #{back} &middot; <a href="/##{location.pathname}" target="_top" class="_error-link">Reload</a>
18 &middot; <a href="#" class="_error-link" data-retry>Retry</a> """18 &middot; <a href="#" class="_error-link" data-retry>Retry</a> """
19 19
@@ -57,9 +57,9 @@ app.templates.unsupportedBrowser = """ · 37 unmodified lines
57 <p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:57 <p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:
58 <ul class="_fail-list">58 <ul class="_fail-list">
59 <li>Recent versions of Firefox, Chrome, or Opera59 <li>Recent versions of Firefox, Chrome, or Opera
60 <li>Safari 9.1+60 <li>Safari 11.1+
61 <li>Edge 16+61 <li>Edge 17+
62 <li>iOS 10+62 <li>iOS 11.3+
63 </ul>63 </ul>
64 <p class="_fail-text">64 <p class="_fail-text">
65 If you're unable to upgrade, we apologize.65 If you're unable to upgrade, we apologize.
assets/javascripts/templates/pages/offline_tmpl.coffee+9−4
@@ -25,8 +25,8 @@ app.templates.offlinePage = (docs) -> """ · 24 unmodified lines
25 <h2 class="_block-heading">Questions & Answers</h2>25 <h2 class="_block-heading">Questions & Answers</h2>
26 <dl>26 <dl>
27 <dt>How does this work?27 <dt>How does this work?
28 <dd>Each page is cached as a key-value pair in <a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API">IndexedDB</a> (downloaded from a single file).<br>28 <dd>Each page is cached as a key-value pair in <a href="https://devdocs.io/dom/indexeddb_api">IndexedDB</a> (downloaded from a single file).<br>
29 The app also uses <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Using_the_application_cache">AppCache</a> and <a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API">localStorage</a> to cache the assets and index files.29 The app also uses <a href="https://devdocs.io/dom/service_worker_api/using_service_workers">Service Workers</a> and <a href="https://devdocs.io/dom/web_storage_api">localStorage</a> to cache the assets and index files.
30 <dt>Can I close the tab/browser?30 <dt>Can I close the tab/browser?
31 <dd>#{canICloseTheTab()}31 <dd>#{canICloseTheTab()}
32 <dt>What if I don't update a documentation?32 <dt>What if I don't update a documentation?
@@ -41,10 +41,15 @@ app.templates.offlinePage = (docs) -> """ · 8 unmodified lines
41"""41"""
42 42
43canICloseTheTab = ->43canICloseTheTab = ->
44 if app.AppCache.isEnabled()44 if app.ServiceWorker.isEnabled()
45 """ Yes! Even offline, you can open a new tab, go to <a href="//devdocs.io">devdocs.io</a>, and everything will work as if you were online (provided you installed all the documentations you want to use beforehand). """45 """ Yes! Even offline, you can open a new tab, go to <a href="//devdocs.io">devdocs.io</a>, and everything will work as if you were online (provided you installed all the documentations you want to use beforehand). """
46 else46 else
47 """ No. AppCache isn't available in your browser (or is disabled), so loading <a href="//devdocs.io">devdocs.io</a> offline won't work.<br>47 reason = "aren't available in your browser (or are disabled)"
48
49 if app.config.env != 'production'
50 reason = "are disabled in your development instance of DevDocs (enable them by setting the <code>ENABLE_SERVICE_WORKER</code> environment variable to <code>true</code>)"
51
52 """ No. Service Workers #{reason}, so loading <a href="//devdocs.io">devdocs.io</a> offline won't work.<br>
48 The current tab will continue to function even when you go offline (provided you installed all the documentations beforehand). """53 The current tab will continue to function even when you go offline (provided you installed all the documentations beforehand). """
49 54
50app.templates.offlineDoc = (doc, status) ->55app.templates.offlineDoc = (doc, status) ->
assets/javascripts/templates/pages/root_tmpl.coffee.erb+2−2
@@ -8,7 +8,7 @@ app.templates.intro = """ · 7 unmodified lines
8 <p>Thanks for downloading DevDocs. Here are a few things you should know:8 <p>Thanks for downloading DevDocs. Here are a few things you should know:
9 <ol class="_intro-list">9 <ol class="_intro-list">
10 <li>Your local version of DevDocs won't self-update. Unless you're modifying the code,10 <li>Your local version of DevDocs won't self-update. Unless you're modifying the code,
11 I&nbsp;recommend using the hosted version at <a href="https://devdocs.io">devdocs.io</a>.11 we&nbsp;recommend using the hosted version at <a href="https://devdocs.io">devdocs.io</a>.
12 <li>Run <code>thor docs:list</code> to see all available documentations.12 <li>Run <code>thor docs:list</code> to see all available documentations.
13 <li>Run <code>thor docs:download &lt;name&gt;</code> to download documentations.13 <li>Run <code>thor docs:download &lt;name&gt;</code> to download documentations.
14 <li>Run <code>thor docs:download --installed</code> to update all downloaded documentations.14 <li>Run <code>thor docs:download --installed</code> to update all downloaded documentations.
@@ -39,7 +39,7 @@ app.templates.intro = """ · 24 unmodified lines
39 <li>DevDocs works <a href="/offline">offline</a>, on mobile, and can be installed on <a href="https://chrome.google.com/webstore/detail/devdocs/mnfehgbmkapmjnhcnbodoamcioleeooe">Chrome</a>.39 <li>DevDocs works <a href="/offline">offline</a>, on mobile, and can be installed on <a href="https://chrome.google.com/webstore/detail/devdocs/mnfehgbmkapmjnhcnbodoamcioleeooe">Chrome</a>.
40 <li>For the latest news, follow <a href="https://twitter.com/DevDocs">@DevDocs</a>.40 <li>For the latest news, follow <a href="https://twitter.com/DevDocs">@DevDocs</a>.
41 <li>DevDocs is free and <a href="https://github.com/freeCodeCamp/devdocs">open source</a>.41 <li>DevDocs is free and <a href="https://github.com/freeCodeCamp/devdocs">open source</a>.
42 <iframe class="_github-btn" src="//ghbtns.com/github-btn.html?user=freeCodeCamp&repo=devdocs&type=watch&count=true" allowtransparency="true" frameborder="0" scrolling="0" width="100" height="20"></iframe>42 <object data="https://img.shields.io/github/stars/freeCodeCamp/devdocs.svg?style=social" type="image/svg+xml" aria-hidden="true"></object>
43 <li>And if you're new to coding, check out <a href="https://www.freecodecamp.org/">freeCodeCamp's open source curriculum</a>.43 <li>And if you're new to coding, check out <a href="https://www.freecodecamp.org/">freeCodeCamp's open source curriculum</a>.
44 </ol>44 </ol>
45 <p>Happy coding!45 <p>Happy coding!
assets/javascripts/views/content/entry_page.coffee+1−1
@@ -123,7 +123,7 @@ class app.views.EntryPage extends app.View · 122 unmodified lines
123 @render @tmpl('pageLoadError')123 @render @tmpl('pageLoadError')
124 @resetClass()124 @resetClass()
125 @addClass @constructor.errorClass125 @addClass @constructor.errorClass
126 app.appCache?.update()126 app.serviceWorker?.update()
127 return127 return
128 128
129 cache: ->129 cache: ->
assets/javascripts/views/content/settings_page.coffee+1−11
@@ -1,7 +1,4 @@
1class app.views.SettingsPage extends app.View1class app.views.SettingsPage extends app.View
2 LAYOUTS = ['_max-width', '_sidebar-hidden', '_native-scrollbars']
3 SIDEBAR_HIDDEN_LAYOUT = '_sidebar-hidden'
4
5 @className: '_static'2 @className: '_static'
6 3
7 @events:4 @events:
@@ -18,25 +15,18 @@ class app.views.SettingsPage extends app.View · 10 unmodified lines
18 settings.smoothScroll = !app.settings.get('fastScroll')15 settings.smoothScroll = !app.settings.get('fastScroll')
19 settings.arrowScroll = app.settings.get('arrowScroll')16 settings.arrowScroll = app.settings.get('arrowScroll')
20 settings.analyticsConsent = app.settings.get('analyticsConsent')17 settings.analyticsConsent = app.settings.get('analyticsConsent')
21 settings[layout] = app.settings.hasLayout(layout) for layout in LAYOUTS18 settings[layout] = app.settings.hasLayout(layout) for layout in app.settings.LAYOUTS
22 settings19 settings
23 20
24 getTitle: ->21 getTitle: ->
25 'Preferences'22 'Preferences'
26 23
27 toggleDark: (enable) ->24 toggleDark: (enable) ->
28 html = document.documentElement
29 html.classList.toggle('_theme-default')
30 html.classList.toggle('_theme-dark')
31 app.settings.set('dark', !!enable)25 app.settings.set('dark', !!enable)
32 app.appCache?.updateInBackground()
33 return26 return
34 27
35 toggleLayout: (layout, enable) ->28 toggleLayout: (layout, enable) ->
36 document.body.classList[if enable then 'add' else 'remove'](layout) unless layout is SIDEBAR_HIDDEN_LAYOUT
37 document.body.classList[if $.overlayScrollbarsEnabled() then 'add' else 'remove']('_overlay-scrollbars')
38 app.settings.setLayout(layout, enable)29 app.settings.setLayout(layout, enable)
39 app.appCache?.updateInBackground()
40 return30 return
41 31
42 toggleSmoothScroll: (enable) ->32 toggleSmoothScroll: (enable) ->
assets/javascripts/views/layout/resizer.coffee+2−9
@@ -11,9 +11,6 @@ class app.views.Resizer extends app.View · 10 unmodified lines
11 init: ->11 init: ->
12 @el.setAttribute('draggable', 'true')12 @el.setAttribute('draggable', 'true')
13 @appendTo $('._app')13 @appendTo $('._app')
14
15 @style = $('style[data-resizer]')
16 @size = @style.getAttribute('data-size')
17 return14 return
18 15
19 MIN = 26016 MIN = 260
@@ -24,15 +21,11 @@ class app.views.Resizer extends app.View · 4 unmodified lines
24 return unless value > 021 return unless value > 0
25 value = Math.min(Math.max(Math.round(value), MIN), MAX)22 value = Math.min(Math.max(Math.round(value), MIN), MAX)
26 newSize = "#{value}px"23 newSize = "#{value}px"
27 @style.innerHTML = @style.innerHTML.replace(new RegExp(@size, 'g'), newSize)24 document.documentElement.style.setProperty('--sidebarWidth', newSize)
28 @size = newSize25 app.settings.setSize(value) if save
29 if save
30 app.settings.setSize(value)
31 app.appCache?.updateInBackground()
32 return26 return
33 27
34 onDragStart: (event) =>28 onDragStart: (event) =>
35 @style.removeAttribute('disabled')
36 event.dataTransfer.effectAllowed = 'link'29 event.dataTransfer.effectAllowed = 'link'
37 event.dataTransfer.setData('Text', '')30 event.dataTransfer.setData('Text', '')
38 $.on(window, 'dragover', @onDrag)31 $.on(window, 'dragover', @onDrag)
assets/javascripts/views/layout/settings.coffee+1−9
@@ -25,7 +25,6 @@ class app.views.Settings extends app.View · 24 unmodified lines
25 if super25 if super
26 @render()26 @render()
27 document.body.classList.remove(SIDEBAR_HIDDEN_LAYOUT)27 document.body.classList.remove(SIDEBAR_HIDDEN_LAYOUT)
28 app.appCache?.on 'progress', @onAppCacheProgress
29 return28 return
30 29
31 deactivate: ->30 deactivate: ->
@@ -33,7 +32,6 @@ class app.views.Settings extends app.View · 1 unmodified line
33 @resetClass()32 @resetClass()
34 @docPicker.detach()33 @docPicker.detach()
35 document.body.classList.add(SIDEBAR_HIDDEN_LAYOUT) if app.settings.hasLayout(SIDEBAR_HIDDEN_LAYOUT)34 document.body.classList.add(SIDEBAR_HIDDEN_LAYOUT) if app.settings.hasLayout(SIDEBAR_HIDDEN_LAYOUT)
36 app.appCache?.off 'progress', @onAppCacheProgress
37 return35 return
38 36
39 render: ->37 render: ->
@@ -52,7 +50,7 @@ class app.views.Settings extends app.View · 12 unmodified lines
52 docs = @docPicker.getSelectedDocs()50 docs = @docPicker.getSelectedDocs()
53 app.settings.setDocs(docs)51 app.settings.setDocs(docs)
54 52
55 @saveBtn.textContent = if app.appCache then 'Downloading\u2026' else 'Saving\u2026'53 @saveBtn.textContent = 'Saving\u2026'
56 disabledDocs = new app.collections.Docs(doc for doc in app.docs.all() when docs.indexOf(doc.slug) is -1)54 disabledDocs = new app.collections.Docs(doc for doc in app.docs.all() when docs.indexOf(doc.slug) is -1)
57 disabledDocs.uninstall ->55 disabledDocs.uninstall ->
58 app.db.migrate()56 app.db.migrate()
@@ -83,9 +81,3 @@ class app.views.Settings extends app.View · 24 unmodified lines
83 $.stopEvent(event)81 $.stopEvent(event)
84 app.router.show '/'82 app.router.show '/'
85 return83 return
86
87 onAppCacheProgress: (event) =>
88 if event.lengthComputable
89 percentage = Math.round event.loaded * 100 / event.total
90 @saveBtn.textContent = "Downloading\u2026 (#{percentage}%)"
91 return
assets/javascripts/views/search/search.coffee+8−0
@@ -30,6 +30,9 @@ class app.views.Search extends app.View · 29 unmodified lines
30 .on 'results', @onResults30 .on 'results', @onResults
31 .on 'end', @onEnd31 .on 'end', @onEnd
32 32
33 @scope
34 .on 'change', @onScopeChange
35
33 app.on 'ready', @onReady36 app.on 'ready', @onReady
34 $.on window, 'hashchange', @searchUrl37 $.on window, 'hashchange', @searchUrl
35 $.on window, 'focus', @onWindowFocus38 $.on window, 'focus', @onWindowFocus
@@ -138,6 +141,11 @@ class app.views.Search extends app.View · 102 unmodified lines
138 $.stopEvent(event)141 $.stopEvent(event)
139 return142 return
140 143
144 onScopeChange: =>
145 @value = ''
146 @onInput()
147 return
148
141 afterRoute: (name, context) =>149 afterRoute: (name, context) =>
142 return if app.shortcuts.eventInProgress?.name is 'escape'150 return if app.shortcuts.eventInProgress?.name is 'escape'
143 @reset(true) if not context.init and app.router.isIndex()151 @reset(true) if not context.init and app.router.isIndex()
assets/javascripts/views/search/search_scope.coffee+23−5
@@ -6,7 +6,9 @@ class app.views.SearchScope extends app.View · 5 unmodified lines
6 tag: '._search-tag'6 tag: '._search-tag'
7 7
8 @events:8 @events:
9 click: 'onClick'
9 keydown: 'onKeydown'10 keydown: 'onKeydown'
11 textInput: 'onTextInput'
10 12
11 @routes:13 @routes:
12 after: 'afterRoute'14 after: 'afterRoute'
@@ -87,17 +89,33 @@ class app.views.SearchScope extends app.View · 74 unmodified lines
87 @trigger 'change', null, previousDoc89 @trigger 'change', null, previousDoc
88 return90 return
89 91
92 doScopeSearch: (event) =>
93 @search @input.value[0...@input.selectionStart]
94 $.stopEvent(event) if @doc
95 return
96
97 onClick: (event) =>
98 if event.target is @tag
99 @reset()
100 $.stopEvent(event)
101 return
102
90 onKeydown: (event) =>103 onKeydown: (event) =>
91 if event.which is 8 # backspace104 if event.which is 8 # backspace
92 if @doc and not @input.value105 if @doc and @input.selectionEnd is 0
93 $.stopEvent(event)
94 @reset()106 @reset()
95 else if not @doc and @input.value107 $.stopEvent(event)
108 else if not @doc and @input.value and not $.isChromeForAndroid()
96 return if event.ctrlKey or event.metaKey or event.altKey or event.shiftKey109 return if event.ctrlKey or event.metaKey or event.altKey or event.shiftKey
97 if event.which is 9 or # tab110 if event.which is 9 or # tab
98 (event.which is 32 and app.isMobile()) # space111 (event.which is 32 and app.isMobile()) # space
99 @search @input.value[0...@input.selectionStart]112 @doScopeSearch(event)
100 $.stopEvent(event) if @doc113 return
114
115 onTextInput: (event) =>
116 return unless $.isChromeForAndroid()
117 if not @doc and @input.value and event.data == ' '
118 @doScopeSearch(event)
101 return119 return
102 120
103 extractHashValue: ->121 extractHashValue: ->
assets/javascripts/views/sidebar/sidebar.coffee+1−1
@@ -28,7 +28,7 @@ class app.views.Sidebar extends app.View · 27 unmodified lines
28 28
29 app.on 'ready', @onReady29 app.on 'ready', @onReady
30 30
31 $.on document.documentElement, 'mouseleave', (event) => @display() if event.clientX < 1031 $.on document.documentElement, 'mouseleave', (event) => @display() unless event.clientX <= 0
32 $.on document.documentElement, 'mouseenter', => @resetDisplay(forceNoHover: false)32 $.on document.documentElement, 'mouseenter', => @resetDisplay(forceNoHover: false)
33 return33 return
34 34
assets/stylesheets/components/_header.scss+1−0
@@ -215,5 +215,6 @@ · 214 unmodified lines
215 color: var(--textColorLight);215 color: var(--textColorLight);
216 background: var(--searchTagBackground);216 background: var(--searchTagBackground);
217 border-radius: 2px;217 border-radius: 2px;
218 cursor: pointer;
218 @extend %truncate-text;219 @extend %truncate-text;
219}220}
docs/adding-docs.md+23−0
@@ -0,0 +1,23 @@
1Adding a documentation may look like a daunting task but once you get the hang of it, it's actually quite simple. Don't hesitate to ask for help [in Gitter](https://gitter.im/FreeCodeCamp/DevDocs) if you ever get stuck.
2
3**Note:** please read the [contributing guidelines](../.github/CONTRIBUTING.md) before submitting a new documentation.
4
51. Create a subclass of `Docs::UrlScraper` or `Docs::FileScraper` in the `lib/docs/scrapers/` directory. Its name should be the [PascalCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename (e.g. `my_doc` → `MyDoc`)
62. Add the appropriate class attributes and filter options (see the [Scraper Reference](./scraper-reference.md) page).
73. Check that the scraper is listed in `thor docs:list`.
84. Create filters specific to the scraper in the `lib/docs/filters/[my_doc]/` directory and add them to the class's [filter stacks](./scraper-reference.md#filter-stacks). You may create any number of filters but will need at least the following two:
9 * A [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter whose task is to clean the HTML markup (e.g. adding `id` attributes to headings) and remove everything superfluous and/or nonessential.
10 * An [`Entries`](./filter-reference.md#entriesfilter) filter whose task is to determine the pages' metadata (the list of entries, each with a name, type and path).
11 The [Filter Reference](./filter-reference.md) page has all the details about filters.
125. Using the `thor docs:page [my_doc] [path]` command, check that the scraper works properly. Files will appear in the `public/docs/[my_doc]/` directory (but not inside the app as the command doesn't touch the index). `path` in this case refers to either the remote path (if using `UrlScraper`) or the local path (if using `FileScraper`).
136. Generate the full documentation using the `thor docs:generate [my_doc] --force` command. Additionally, you can use the `--verbose` option to see which files are being created/updated/deleted (useful to see what changed since the last run), and the `--debug` option to see which URLs are being requested and added to the queue (useful to pin down which page adds unwanted URLs to the queue).
147. Start the server, open the app, enable the documentation, and see how everything plays out.
158. Tweak the scraper/filters and repeat 5) and 6) until the pages and metadata are ok.
169. To customize the pages' styling, create an SCSS file in the `assets/stylesheets/pages/` directory and import it in both `application.css.scss` AND `application-dark.css.scss`. Both the file and CSS class should be named `_[type]` where [type] is equal to the scraper's `type` attribute (documentations with the same type share the same custom CSS and JS). Setting the type to `simple` will apply the general styling rules in `assets/stylesheets/pages/_simple.scss`, which can be used for documentations where little to no CSS changes are needed.
1710. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works).
1811. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. It'll be added to the icon spritesheet after your pull request is merged.
1912. Add the documentation's copyright details to the list in `assets/javascripts/templates/pages/about_tmpl.coffee`. This is the data shown in the table on the [about](https://devdocs.io/about) page, and is ordered alphabetically. Simply copying an existing item, placing it in the right slot and updating the values to match the new scraper will do the job.
20
21If the documentation includes more than a few hundreds pages and is available for download, try to scrape it locally (e.g. using `FileScraper`). It'll make the development process much faster and avoids putting too much load on the source site. (It's not a problem if your scraper is coupled to your local setup, just explain how it works in your pull request.)
22
23Finally, try to document your scraper and filters' behavior as much as possible using comments (e.g. why some URLs are ignored, HTML markup removed, metadata that way, etc.). It'll make updating the documentation much easier.
docs/filter-reference.md+224−0
@@ -0,0 +1,224 @@
1**Table of contents:**
2
3* [Overview](#overview)
4* [Instance methods](#instance-methods)
5* [Core filters](#core-filters)
6* [Custom filters](#custom-filters)
7 - [CleanHtmlFilter](#cleanhtmlfilter)
8 - [EntriesFilter](#entriesfilter)
9
10## Overview
11
12Filters use the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. They take an HTML string or [Nokogiri](http://nokogiri.org/) node as input, optionally perform modifications and/or extract information from it, and then outputs the result. Together they form a pipeline where each filter hands its output to the next filter's input. Every documentation page passes through this pipeline before being copied on the local filesystem.
13
14Filters are subclasses of the [`Docs::Filter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter.rb) class and require a `call` method. A basic implementation looks like this:
15
16```ruby
17module Docs
18 class CustomFilter < Filter
19 def call
20 doc
21 end
22 end
23end
24```
25
26Filters which manipulate the Nokogiri node object (`doc` and related methods) are _HTML filters_ and must not manipulate the HTML string (`html`). Vice-versa, filters which manipulate the string representation of the document are _text filters_ and must not manipulate the Nokogiri node object. The two types are divided into two stacks within the scrapers. These stacks are then combined into a pipeline that calls the HTML filters before the text filters (more details [here](./scraper-reference.md#filter-stacks)). This is to avoid parsing the document multiple times.
27
28The `call` method must return either `doc` or `html`, depending on the type of filter.
29
30## Instance methods
31
32* `doc` [Nokogiri::XML::Node]
33 The Nokogiri representation of the container element.
34 See [Nokogiri's API docs](http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Node) for the list of available methods.
35
36* `html` [String]
37 The string representation of the container element.
38
39* `context` [Hash] **(frozen)**
40 The scraper's `options` along with a few additional keys: `:base_url`, `:root_url`, `:root_page` and `:url`.
41
42* `result` [Hash]
43 Used to store the page's metadata and pass back information to the scraper.
44 Possible keys:
45
46 - `:path` — the page's normalized path
47 - `:store_path` — the path where the page will be stored (equal to `:path` with `.html` at the end)
48 - `:internal_urls` — the list of distinct internal URLs found within the page
49 - `:entries` — the [`Entry`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/entry.rb) objects to add to the index
50
51* `css`, `at_css`, `xpath`, `at_xpath`
52 Shortcuts for `doc.css`, `doc.xpath`, etc.
53
54* `base_url`, `current_url`, `root_url` [Docs::URL]
55 Shortcuts for `context[:base_url]`, `context[:url]`, and `context[:root_url]` respectively.
56
57* `root_path` [String]
58 Shortcut for `context[:root_path]`.
59
60* `subpath` [String]
61 The sub-path from the base URL of the current URL.
62 _Example: if `base_url` equals `example.com/docs` and `current_url` equals `example.com/docs/file?raw`, the returned value is `/file`._
63
64* `slug` [String]
65 The `subpath` removed of any leading slash or `.html` extension.
66 _Example: if `subpath` equals `/dir/file.html`, the returned value is `dir/file`._
67
68* `root_page?` [Boolean]
69 Returns `true` if the current page is the root page.
70
71* `initial_page?` [Boolean]
72 Returns `true` if the current page is the root page or its subpath is one of the scraper's `initial_paths`.
73
74## Core filters
75
76* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
77* [`CleanHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
78* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
79* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
80* [`NormalizePathsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
81* [`CleanLocalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
82* [`InnerHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
83* [`CleanTextFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
84* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
85* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb) — prepends the document with a title (disabled by default)
86* [`EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) — abstract filter for extracting the page's metadata
87
88## Custom filters
89
90Scrapers can have any number of custom filters but require at least the two described below.
91
92**Note:** filters are located in the [`lib/docs/filters`](https://github.com/Thibaut/devdocs/tree/master/lib/docs/filters/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
93
94### `CleanHtmlFilter`
95
96The `CleanHtml` filter is tasked with cleaning the HTML markup where necessary and removing anything superfluous or nonessential. Only the core documentation should remain at the end.
97
98Nokogiri's many jQuery-like methods make it easy to search and modify elements — see the [API docs](http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Node).
99
100Here's an example implementation that covers the most common use-cases:
101
102```ruby
103module Docs
104 class MyScraper
105 class CleanHtmlFilter < Filter
106 def call
107 css('hr').remove
108 css('#changelog').remove if root_page?
109
110 # Set id attributes on <h3> instead of an empty <a>
111 css('h3').each do |node|
112 node['id'] = node.at_css('a')['id']
113 end
114
115 # Make proper table headers
116 css('td.header').each do |node|
117 node.name = 'th'
118 end
119
120 # Remove code highlighting
121 css('pre').each do |node|
122 node.content = node.content
123 end
124
125 doc
126 end
127 end
128 end
129end
130```
131
132**Notes:**
133
134* Empty elements will be automatically removed by the core `CleanTextFilter` later in the pipeline's execution.
135* Although the goal is to end up with a clean version of the page, try to keep the number of modifications to a minimum, so as to make the code easier to maintain. Custom CSS is the preferred way of normalizing the pages (except for hiding stuff which should always be done by removing the markup).
136* Try to document your filter's behavior as much as possible, particularly modifications that apply only to a subset of pages. It'll make updating the documentation easier.
137
138### `EntriesFilter`
139
140The `Entries` filter is responsible for extracting the page's metadata, represented by a set of _entries_, each with a name, type and path.
141
142The following two models are used under the hood to represent the metadata:
143
144* [`Entry(name, type, path)`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/entry.rb)
145* [`Type(name, slug, count)`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/models/type.rb)
146
147Each scraper must implement its own `EntriesFilter` by subclassing the [`Docs::EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) class. The base class already implements the `call` method and includes four methods which the subclasses can override:
148
149* `get_name` [String]
150 The name of the default entry (aka. the page's name).
151 It is usually guessed from the `slug` (documented above) or by searching the HTML markup.
152 **Default:** modified version of `slug` (underscores are replaced with spaces and forward slashes with dots)
153
154* `get_type` [String]
155 The type of the default entry (aka. the page's type).
156 Entries without a type can be searched for but won't be listed in the app's sidebar (unless no other entries have a type).
157 **Default:** `nil`
158
159* `include_default_entry?` [Boolean]
160 Whether to include the default entry.
161 Used when a page consists of multiple entries (returned by `additional_entries`) but doesn't have a name/type of its own, or to remove a page from the index (if it has no additional entries), in which case it won't be copied on the local filesystem and any link to it in the other pages will be broken (as explained on the [Scraper Reference](./scraper-reference.md) page, this is used to keep the `:skip` / `:skip_patterns` options to a maintainable size, or if the page includes links that can't reached from anywhere else).
162 **Default:** `true`
163
164* `additional_entries` [Array]
165 The list of additional entries.
166 Each entry is represented by an Array of three attributes: its name, fragment identifier, and type. The fragment identifier refers to the `id` attribute of the HTML element (usually a heading) that the entry relates to. It is combined with the page's path to become the entry's path. If absent or `nil`, the page's path is used. If the type is absent or `nil`, the default `type` is used.
167 Example: `[ ['One'], ['Two', 'id'], ['Three', nil, 'type'] ]` adds three additional entries, the first one named "One" with the default path and type, the second one named "Two" with the URL fragment "#id" and the default type, and the third one named "Three" with the default path and the type "type".
168 The list is usually constructed by running through the markup. Exceptions can also be hard-coded for specific pages.
169 **Default:** `[]`
170
171The following accessors are also available, but must not be overridden:
172
173* `name` [String]
174 Memoized version of `get_name` (`nil` for the root page).
175
176* `type` [String]
177 Memoized version of `get_type` (`nil` for the root page).
178
179**Notes:**
180
181* Leading and trailing whitespace is automatically removed from names and types.
182* Names must be unique across the documentation and as short as possible (ideally less than 30 characters). Whenever possible, methods should be differentiated from properties by appending `()`, and instance methods should be differentiated from class methods using the `Class#method` or `object.method` conventions.
183* You can call `name` from `get_type` or `type` from `get_name` but doing both will cause a stack overflow (i.e. you can infer the name from the type or the type from the name, but you can't do both at the same time). Don't call `get_name` or `get_type` directly as their value isn't memoized.
184* The root page has no name and no type (both are `nil`). `get_name` and `get_type` won't get called with the page (but `additional_entries` will).
185* `Docs::EntriesFilter` is an _HTML filter_. It must be added to the scraper's `html_filters` stack.
186* Try to document the code as much as possible, particularly the special cases. It'll make updating the documentation easier.
187
188**Example:**
189
190```ruby
191module Docs
192 class MyScraper
193 class EntriesFilter < Docs::EntriesFilter
194 def get_name
195 node = at_css('h1')
196 result = node.content.strip
197 result << ' event' if type == 'Events'
198 result << '()' if node['class'].try(:include?, 'function')
199 result
200 end
201
202 def get_type
203 object, method = *slug.split('/')
204 method ? object : 'Miscellaneous'
205 end
206
207 def additional_entries
208 return [] if root_page?
209
210 css('h2').map do |node|
211 [node.content, node['id']]
212 end
213 end
214
215 def include_default_entry?
216 !at_css('.obsolete')
217 end
218 end
219 end
220end
221```
222
223
224return [[Home]]
docs/maintainers.mdrenamed from docs/maintainers_guide.md+7−7
@@ -4,7 +4,7 @@ This document is intended for [DevDocs maintainers](#list-of-maintainers). · 3 unmodified lines
4 4
5## Merging pull requests5## Merging pull requests
6 6
7- Unless the change is trivial or in an area that you are familiar with, PRs should be approved by at least two maintainers before being merged.7- PRs should be approved by at least one maintainer before being merged.
8 8
9- PRs that add or update documentations should always be merged locally, and the app deployed, before the merge is pushed to GitHub.9- PRs that add or update documentations should always be merged locally, and the app deployed, before the merge is pushed to GitHub.
10 10
@@ -21,8 +21,7 @@ The process for updating docs is as follow: · 10 unmodified lines
21- Commit the changes (protip: use the `thor docs:commit` command documented below).21- Commit the changes (protip: use the `thor docs:commit` command documented below).
22- Optional: do more updates.22- Optional: do more updates.
23- Run `thor docs:upload` (documented below).23- Run `thor docs:upload` (documented below).
24- [Deploy the app](#deploying-devdocs) and verify that everything works in production.24- Push to GitHub to [deploy the app](#deploying-devdocs) and verify that everything works in production.
25- Push to GitHub.
26- Run `thor docs:clean` (documented below).25- Run `thor docs:clean` (documented below).
27 26
28Note: changes to `public/docs/docs.json` should never be committed. This file reflects which documentations have been downloaded or generated locally, which is always none on a fresh `git clone`.27Note: changes to `public/docs/docs.json` should never be committed. This file reflects which documentations have been downloaded or generated locally, which is always none on a fresh `git clone`.
@@ -82,21 +81,22 @@ In addition to the [publicly-documented commands](https://github.com/freeCodeCam · 53 unmodified lines
82 81
83## Deploying DevDocs82## Deploying DevDocs
84 83
85Once docs have been uploaded via `thor docs:upload` (if applicable), deploying DevDocs is as simple as running `git push heroku master`. See [Heroku's documentation](https://devcenter.heroku.com/articles/git) for more information.84Once docs have been uploaded via `thor docs:upload` (if applicable), you can push to the DevDocs master branch (or merge the PR containing the updates). If the Travis build succeeds, the Heroku application will be deployed automatically.
86 85
87- If you're deploying documentation updates, verify that the documentations work properly once the deploy is done (you will need to reload [devdocs.io](https://devdocs.io/) a couple times for the application cache to update and the new version to load).86- If you're deploying documentation updates, verify that the documentations work properly once the deploy is done. Keep in mind that you'll need to wait a few seconds for the service worker to finish caching the new assets. You should see a "DevDocs has been updated" notification appear when the caching is done, after which you need to refresh the page to see the changes.
88- If you're deploying frontend changes, monitor [Sentry](https://sentry.io/devdocs/devdocs-js/) for new JS errors once the deploy is done.87- If you're deploying frontend changes, monitor [Sentry](https://sentry.io/devdocs/devdocs-js/) for new JS errors once the deploy is done.
89- If you're deploying server changes, monitor New Relic (accessible through [the Heroku dashboard](https://dashboard.heroku.com/apps/devdocs)) for Ruby exceptions and throughput or response time changes once the deploy is done.88- If you're deploying server changes, monitor New Relic (accessible through [the Heroku dashboard](https://dashboard.heroku.com/apps/devdocs)) for Ruby exceptions and throughput or response time changes once the deploy is done.
90 89
91If any issue arises, run `heroku rollback` to rollback to the previous of the app (this can also be done via Heroku's UI). Note that this will not revert changes made to documentation files that were uploaded via `thor docs:upload`. Try and fix the issue as quickly as possible, then re-deploy the app. Reach out to other maintainers if you need help.90If any issue arises, run `heroku rollback` to rollback to the previous version of the app (this can also be done via Heroku's UI). Note that this will not revert changes made to documentation files that were uploaded via `thor docs:upload`. Try and fix the issue as quickly as possible, then re-deploy the app. Reach out to other maintainers if you need help.
92 91
93If this is your first deploy, make sure another maintainer is around to assist. 92If this is your first deploy, make sure another maintainer is around to assist.
94 93
95## List of maintainers94## List of maintainers
96 95
97- [Beau Carnes](https://github.com/beaucarnes)
98- [Jed Fox](https://github.com/j-f1)96- [Jed Fox](https://github.com/j-f1)
99- [Jasper van Merle](https://github.com/jmerle)97- [Jasper van Merle](https://github.com/jmerle)
98- [Ahmad Abdolsaheb](https://github.com/ahmadabdolsaheb)
99- [Mrugesh Mohapatra](https://github.com/raisedadead)
100- [Thibaut Courouble](https://github.com/thibaut)100- [Thibaut Courouble](https://github.com/thibaut)
101 101
102Interested in helping maintain DevDocs? Come talk to us on [Gitter](https://gitter.im/FreeCodeCamp/DevDocs) :)102Interested in helping maintain DevDocs? Come talk to us on [Gitter](https://gitter.im/FreeCodeCamp/DevDocs) :)
docs/scraper-reference.md+186−0
@@ -0,0 +1,186 @@
1**Table of contents:**
2
3* [Overview](#overview)
4* [Configuration](#configuration)
5 - [Attributes](#attributes)
6 - [Filter stacks](#filter-stacks)
7 - [Filter options](#filter-options)
8
9## Overview
10
11Starting from a root URL, scrapers recursively follow links that match a set of rules, passing each valid response through a chain of filters before writing the file on the local filesystem. They also create an index of the pages' metadata (determined by one filter), which is dumped into a JSON file at the end.
12
13Scrapers rely on the following libraries:
14
15* [Typhoeus](https://github.com/typhoeus/typhoeus) for making HTTP requests
16* [HTML::Pipeline](https://github.com/jch/html-pipeline) for applying filters
17* [Nokogiri](http://nokogiri.org/) for parsing HTML
18
19There are currently two kinds of scrapers: [`UrlScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/url_scraper.rb) which downloads files via HTTP and [`FileScraper`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/scrapers/file_scraper.rb) which reads them from the local filesystem. They function almost identically (both use URLs), except that `FileScraper` substitutes the base URL with a local path before reading a file. `FileScraper` uses the placeholder `localhost` base URL by default and includes a filter to remove any URL pointing to it at the end.
20
21To be processed, a response must meet the following requirements:
22
23* 200 status code
24* HTML content type
25* effective URL (after redirection) contained in the base URL (explained below)
26
27(`FileScraper` only checks if the file exists and is not empty.)
28
29Each URL is requested only once (case-insensitive).
30
31## Configuration
32
33Configuration is done via class attributes and divided into three main categories:
34
35* [Attributes](#attributes) — essential information such as name, version, URL, etc.
36* [Filter stacks](#filter-stacks) — the list of filters that will be applied to each page.
37* [Filter options](#filter-options) — the options passed to said filters.
38
39**Note:** scrapers are located in the [`lib/docs/scrapers`](https://github.com/Thibaut/devdocs/tree/master/lib/docs/scrapers/) directory. The class's name must be the [CamelCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename.
40
41### Attributes
42
43* `name` [String]
44 Must be unique.
45 Defaults to the class's name.
46
47* `slug` [String]
48 Must be unique, lowercase, and not include dashes (underscores are ok).
49 Defaults to `name` lowercased.
50
51* `type` [String] **(required, inherited)**
52 Defines the CSS class name (`_[type]`) and custom JavaScript class (`app.views.[Type]Page`) that will be added/loaded on each page. Documentations sharing a similar structure (e.g. generated with the same tool or originating from the same website) should use the same `type` to avoid duplicating the CSS and JS.
53 Must include lowercase letters only.
54
55* `release` [String] **(required)**
56 The version of the software at the time the scraper was last run. This is only informational and doesn't affect the scraper's behavior.
57
58* `base_url` [String] **(required in `UrlScraper`)**
59 The documents' location. Only URLs _inside_ the `base_url` will be scraped. "inside" more or less means "starting with" except that `/docs` is outside `/doc` (but `/doc/` is inside).
60 Defaults to `localhost` in `FileScraper`. _(Note: any iframe, image, or skipped link pointing to localhost will be removed by the `CleanLocalUrls` filter; the value should be overridden if the documents are available online.)_
61 Unless `root_path` is set, the root/initial URL is equal to `base_url`.
62
63* `root_path` [String] **(inherited)**
64 The path from the `base_url` of the root URL.
65
66* `initial_paths` [Array] **(inherited)**
67 A list of paths (from the `base_url`) to add to the initial queue. Useful for scraping isolated documents.
68 Defaults to `[]`. _(Note: the `root_path` is added to the array at runtime.)_
69
70* `dir` [String] **(required, `FileScraper` only)**
71 The absolute path where the files are located on the local filesystem.
72 _Note: `FileScraper` works exactly like `UrlScraper` (manipulating the same kind of URLs) except that it substitutes `base_url` with `dir` in order to read files instead of making HTTP requests._
73
74* `params` [Hash] **(inherited, `UrlScraper` only)**
75 Query string parameters to append to every URL. (e.g. `{ format: 'raw' }` → `?format=raw`)
76 Defaults to `{}`.
77
78* `abstract` [Boolean]
79 Make the scraper abstract / not runnable. Used for sharing behavior with other scraper classes (e.g. all MDN scrapers inherit from the abstract [`Mdn`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/scrapers/mdn/mdn.rb) class).
80 Defaults to `false`.
81
82### Filter stacks
83
84Each scraper has two [filter](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter.rb) [stacks](https://github.com/Thibaut/devdocs/blob/master/lib/docs/core/filter_stack.rb): `html_filters` and `text_filters`. They are combined into a pipeline (using the [HTML::Pipeline](https://github.com/jch/html-pipeline) library) which causes each filter to hand its output to the next filter's input.
85
86HTML filters are executed first and manipulate a parsed version of the document (a [Nokogiri](http://nokogiri.org/Nokogiri/XML/Node.html) node object), whereas text filters manipulate the document as a string. This separation avoids parsing the document multiple times.
87
88Filter stacks are like sorted sets. They can modified using the following methods:
89
90```ruby
91push(*names) # append one or more filters at the end
92insert_before(index, *names) # insert one filter before another (index can be a name)
93insert_after(index, *names) # insert one filter after another (index can be a name)
94replace(index, name) # replace one filter with another (index can be a name)
95```
96
97"names" are `require` paths relative to `Docs` (e.g. `jquery/clean_html` → `Docs::Jquery::CleanHtml`).
98
99Default `html_filters`:
100
101* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside)
102* [`CleanHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `<script>`, `<style>`, etc.
103* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb) — replaces all URLs with their fully qualified counterpart
104* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb) — detects internal URLs (the ones to scrape) and replaces them with their unqualified, relative counterpart
105* [`NormalizePathsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_paths.rb) — makes the internal paths consistent (e.g. always end with `.html`)
106* [`CleanLocalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_local_urls.rb) — removes links, iframes and images pointing to localhost (`FileScraper` only)
107
108Default `text_filters`:
109
110* [`InnerHtmlFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/inner_html.rb) — converts the document to a string
111* [`CleanTextFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/clean_text.rb) — removes empty nodes
112* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb) — appends the license info and link to the original document
113
114Additionally:
115
116* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb) is a core HTML filter, disabled by default, which prepends the document with a title (`<h1>`).
117* [`EntriesFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/entries.rb) is an abstract HTML filter that each scraper must implement and responsible for extracting the page's metadata.
118
119### Filter options
120
121The filter options are stored in the `options` Hash. The Hash is inheritable (a recursive copy) and empty by default.
122
123More information about how filters work is available on the [Filter Reference](./filter-reference.md) page.
124
125* [`ContainerFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/container.rb)
126
127 - `:container` [String or Proc]
128 A CSS selector of the container element. Everything outside of it will be removed and become unavailable to the other filters. If more than one element match the selector, the first one inside the DOM is used. If no elements match the selector, an error is raised.
129 If the value is a Proc, it is called for each page with the filter instance as argument, and should return a selector or `nil`.
130 The default container is the `<body>` element.
131 _Note: links outside of the container element will not be followed by the scraper. To remove links that should be followed, use a [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter later in the stack._
132
133* [`NormalizeUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/normalize_urls.rb)
134 The following options are used to modify URLs in the pages. They are useful to remove duplicates (when the same page is accessible from multiple URLs) and fix websites that have a bunch of redirections in place (when URLs that should be scraped, aren't, because they are behind a redirection which is outside of the `base_url` — see the MDN scrapers for examples of this).
135
136 - `:replace_urls` [Hash]
137 Replaces all instances of a URL with another.
138 Format: `{ 'original_url' => 'new_url' }`
139 - `:replace_paths` [Hash]
140 Replaces all instances of a sub-path (path from the `base_url`) with another.
141 Format: `{ 'original_path' => 'new_path' }`
142 - `:fix_urls` [Proc]
143 Called with each URL. If the returned value is `nil`, the URL isn't modified. Otherwise the returned value is used as replacement.
144
145 _Note: before these rules are applied, all URLs are converted to their fully qualified counterpart (http://...)._
146
147* [`InternalUrlsFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/internal_urls.rb)
148
149 Internal URLs are the ones _inside_ the scraper's `base_url` ("inside" more or less means "starting with", except that `/docs` is outside `/doc`). They will be scraped unless excluded by one of the following rules. All internal URLs are converted to relative URLs inside the pages.
150
151 - `:skip_links` [Boolean or Proc]
152 If `false`, does not convert or follow any internal URL (creating a single-page documentation).
153 If the value is a Proc, it is called for each page with the filter instance as argument.
154 - `:follow_links` [Proc]
155 Called for page with the filter instance as argument. If the returned value is `false`, does not add internal URLs to the queue.
156 - `:trailing_slash` [Boolean]
157 If `true`, adds a trailing slash to all internal URLs. If `false`, removes it.
158 This is another option used to remove duplicate pages.
159 - `:skip` [Array]
160 Ignores internal URLs whose sub-paths (path from the `base_url`) are in the Array (case-insensitive).
161 - `:skip_patterns` [Array]
162 Ignores internal URLs whose sub-paths match any Regexp in the Array.
163 - `:only` [Array]
164 Ignores internal URLs whose sub-paths aren't in the Array (case-insensitive) and don't match any Regexp in `:only_patterns`.
165 - `:only_patterns` [Array]
166 Ignores internal URLs whose sub-paths don't match any Regexp in the Array and aren't in `:only`.
167
168 If the scraper has a `root_path`, the empty and `/` paths are automatically skipped.
169 If `:only` or `:only_patterns` is set, the root path is automatically added to `:only`.
170
171 _Note: pages can be excluded from the index based on their content using the [`Entries`](./filter-reference.md#entriesfilter) filter. However, their URLs will still be converted to relative in the other pages and trying to open them will return a 404 error. Although not ideal, this is often better than having to maintain a long list of `:skip` URLs._
172
173* [`AttributionFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/attribution.rb)
174
175 - `:attribution` [String] **(required)**
176 An HTML string with the copyright and license information. See the other scrapers for examples.
177
178* [`TitleFilter`](https://github.com/Thibaut/devdocs/blob/master/lib/docs/filters/core/title.rb)
179
180 - `:title` [String or Boolean or Proc]
181 Unless the value is `false`, adds a title to every page.
182 If the value is `nil`, the title is the name of the page as determined by the [`Entries`](./filter-reference.md#entriesfilter) filter. Otherwise the title is the String or the value returned by the Proc (called for each page, with the filter instance as argument). If the Proc returns `nil` or `false`, no title is added.
183 - `:root_title` [String or Boolean]
184 Overrides the `:title` option for the root page only.
185
186 _Note: this filter is disabled by default._
lib/app.rb+35−20
@@ -37,6 +37,9 @@ class App < Sinatra::Application · 36 unmodified lines
37 37
38 set :csp, false38 set :csp, false
39 39
40 require 'docs'
41 Docs.generate_manifest
42
40 Dir[docs_path, root.join(assets_prefix, '*/')].each do |path|43 Dir[docs_path, root.join(assets_prefix, '*/')].each do |path|
41 sprockets.append_path(path)44 sprockets.append_path(path)
42 end45 end
@@ -192,35 +195,47 @@ class App < Sinatra::Application · 149 unmodified lines
192 request.query_string.empty? ? nil : "?#{request.query_string}"195 request.query_string.empty? ? nil : "?#{request.query_string}"
193 end196 end
194 197
195 def manifest_asset_urls198 def service_worker_asset_urls
196 @@manifest_asset_urls ||= [199 @@service_worker_asset_urls ||= [
197 javascript_path('application', asset_host: false),200 javascript_path('application', asset_host: false),
198 stylesheet_path('application'),201 stylesheet_path('application'),
199 image_path('docs-1.png'),202 image_path('docs-1.png'),
200 image_path('docs-1@2x.png'),203 image_path('docs-1@2x.png'),
201 image_path('docs-2.png'),204 image_path('docs-2.png'),
202 image_path('docs-2@2x.png'),205 image_path('docs-2@2x.png'),
203 asset_path('docs.js')206 asset_path('docs.js'),
204 ]207 App.production? ? nil : javascript_path('debug'),
208 ].compact
205 end209 end
206 210
207 def app_size211 # Returns a cache name for the service worker to use which changes if any of the assets changes
208 @app_size ||= memoized_cookies['size'].nil? ? '20rem' : "#{memoized_cookies['size']}px"212 # When a manifest exist, this name is only created once based on the asset manifest because it never changes without a server restart
209 end213 # If a manifest does not exist, it is created every time this method is called because the assets can change while the server is running
214 def service_worker_cache_name
215 if File.exist?(App.assets_manifest_path)
216 if defined?(@@service_worker_cache_name)
217 return @@service_worker_cache_name
218 end
210 219
211 def app_layout220 digest = Sprockets::Manifest
212 memoized_cookies['layout']221 .new(nil, App.assets_manifest_path)
213 end222 .files
223 .values
224 .map {|file| file["digest"]}
225 .join
214 226
215 def app_theme227 return @@service_worker_cache_name ||= Digest::MD5.hexdigest(digest)
216 @app_theme ||= memoized_cookies['dark'].nil? ? 'default' : 'dark'228 else
217 end229 paths = App.sprockets
230 .each_file
231 .to_a
232 .reject {|file| file.start_with?(App.docs_path)}
218 233
219 def dark_theme?234 return App.sprockets.pack_hexdigest(App.sprockets.files_digest(paths))
220 app_theme == 'dark'235 end
221 end236 end
222 237
223 def redirect_via_js(path) # courtesy of HTML5 App Cache238 def redirect_via_js(path)
224 response.set_cookie :initial_path, value: path, expires: Time.now + 15, path: '/'239 response.set_cookie :initial_path, value: path, expires: Time.now + 15, path: '/'
225 redirect '/', 302240 redirect '/', 302
226 end241 end
@@ -243,15 +258,15 @@ class App < Sinatra::Application · 16 unmodified lines
243 end258 end
244 end259 end
245 260
246 get '/manifest.appcache' do261 get '/service-worker.js' do
247 content_type 'text/cache-manifest'262 content_type 'application/javascript'
248 expires 0, :'no-cache'263 expires 0, :'no-cache'
249 erb :manifest264 erb :'service-worker.js'
250 end265 end
251 266
252 get '/' do267 get '/' do
253 return redirect "/#q=#{params[:q]}" if params[:q]268 return redirect "/#q=#{params[:q]}" if params[:q]
254 return redirect '/' unless request.query_string.empty? # courtesy of HTML5 App Cache269 return redirect '/' unless request.query_string.empty?
255 response.headers['Content-Security-Policy'] = settings.csp if settings.csp270 response.headers['Content-Security-Policy'] = settings.csp if settings.csp
256 erb :index271 erb :index
257 end272 end
public/docs/docs.json+0−1
@@ -1 +0,0 @@
1[]
public/images/webapp-icon-192.png+0−0

Binary file not shown.

public/manifest.json+6−0
@@ -4,6 +4,7 @@ · 3 unmodified lines
4 "description": "API Documentation Browser",4 "description": "API Documentation Browser",
5 "start_url": "/",5 "start_url": "/",
6 "display": "standalone",6 "display": "standalone",
7 "background_color": "#EEEEEE",
7 "icons": [8 "icons": [
8 {9 {
9 "src": "/images/webapp-icon-32.png",10 "src": "/images/webapp-icon-32.png",
@@ -25,6 +26,11 @@ · 15 unmodified lines
25 "sizes": "128x128",26 "sizes": "128x128",
26 "type": "image/png"27 "type": "image/png"
27 },28 },
29 {
30 "src": "/images/webapp-icon-192.png",
31 "sizes": "192x192",
32 "type": "image/png"
33 },
28 {34 {
29 "src": "/images/webapp-icon-256.png",35 "src": "/images/webapp-icon-256.png",
30 "sizes": "256x256",36 "sizes": "256x256",
test/app_test.rb+0−81
@@ -43,35 +43,6 @@ class AppTest < MiniTest::Spec · 42 unmodified lines
43 assert last_response.redirect?43 assert last_response.redirect?
44 assert_equal 'https://example.org/', last_response['Location']44 assert_equal 'https://example.org/', last_response['Location']
45 end45 end
46
47 it "sets default size" do
48 get '/'
49 assert_includes last_response.body, 'data-size="20rem"'
50 end
51
52 it "sets size from cookie" do
53 set_cookie('size=42')
54 get '/'
55 assert_includes last_response.body, 'data-size="42px"'
56 end
57
58 it "sets layout from cookie" do
59 set_cookie('layout=foo')
60 get '/'
61 assert_includes last_response.body, '<body class="foo">'
62 end
63
64 it "sets the <html> theme from cookie" do
65 get '/'
66 assert_match %r{<html [^>]*class="[^\"]*_theme-default}, last_response.body
67 refute_includes last_response.body, '_theme-dark'
68
69 set_cookie('dark=1')
70
71 get '/'
72 assert_match %r{<html [^>]*class="[^\"]*_theme-dark}, last_response.body
73 refute_includes last_response.body, '_theme-default'
74 end
75 end46 end
76 47
77 describe "/[static-page]" do48 describe "/[static-page]" do
@@ -106,58 +77,6 @@ class AppTest < MiniTest::Spec · 28 unmodified lines
106 end77 end
107 end78 end
108 79
109 describe "/manifest.appcache" do
110 it "works" do
111 get '/manifest.appcache'
112 assert last_response.ok?
113 end
114
115 it "works with cookie" do
116 set_cookie('docs=css/html~5')
117 get '/manifest.appcache'
118 assert last_response.ok?
119 assert_includes last_response.body, '/css/index.json?1420139788'
120 assert_includes last_response.body, '/html~5/index.json?1420139791'
121 end
122
123 it "ignores invalid docs in the cookie" do
124 set_cookie('docs=foo')
125 get '/manifest.appcache'
126 assert last_response.ok?
127 refute_includes last_response.body, 'foo'
128 end
129
130 it "has the word 'default' when no 'dark' cookie is set" do
131 get '/manifest.appcache'
132 assert_includes last_response.body, '# default'
133 refute_includes last_response.body, '# dark'
134 end
135
136 it "has the word 'dark' when the cookie is set" do
137 set_cookie('dark=1')
138 get '/manifest.appcache'
139 assert_includes last_response.body, '# dark'
140 refute_includes last_response.body, '# default'
141 end
142
143 it "sets default size" do
144 get '/manifest.appcache'
145 assert_includes last_response.body, '20rem'
146 end
147
148 it "sets size from cookie" do
149 set_cookie('size=42')
150 get '/manifest.appcache'
151 assert_includes last_response.body, '42px'
152 end
153
154 it "sets layout from cookie" do
155 set_cookie('layout=foo_layout')
156 get '/manifest.appcache'
157 assert_includes last_response.body, 'foo_layout'
158 end
159 end
160
161 describe "/[doc]" do80 describe "/[doc]" do
162 it "renders when the doc exists and isn't enabled" do81 it "renders when the doc exists and isn't enabled" do
163 set_cookie('docs=html~5')82 set_cookie('docs=html~5')
views/app.erb+1−13
@@ -28,13 +28,7 @@ · 27 unmodified lines
28 </nav>28 </nav>
29 </header>29 </header>
30 <section class="_sidebar" tabindex="-1">30 <section class="_sidebar" tabindex="-1">
31 <div class="_list" role="navigation">31 <div class="_list" role="navigation"></div>
32 <% unless @doc %>
33 <% App.docs.each do |slug, doc| %>
34 <a href="/<%= slug %>/" class="_list-item"><%= doc['full_name'] %></a>
35 <% end %>
36 <% end %>
37 </div>
38 </section>32 </section>
39 <div class="_container" role="document">33 <div class="_container" role="document">
40 <main class="_content _content-loading" role="main"></main>34 <main class="_content _content-loading" role="main"></main>
@@ -63,9 +57,3 @@ · 22 unmodified lines
63 <symbol id="icon-external-link" viewBox="0 0 24 24"><path d="M19 19H5V5h7V3H5c-1.11 0-2 .9-2 2v14c0 1.1.89 2 2 2h14c1.1 0 2-.9 2-2v-7h-2v7zM14 3v2h3.59l-9.83 9.83 1.41 1.41L19 6.41V10h2V3h-7z"/></symbol>57 <symbol id="icon-external-link" viewBox="0 0 24 24"><path d="M19 19H5V5h7V3H5c-1.11 0-2 .9-2 2v14c0 1.1.89 2 2 2h14c1.1 0 2-.9 2-2v-7h-2v7zM14 3v2h3.59l-9.83 9.83 1.41 1.41L19 6.41V10h2V3h-7z"/></symbol>
64 </defs>58 </defs>
65</svg>59</svg>
66<style data-size="<%= app_size %>" data-resizer>
67 ._container { margin-left: <%= app_size %>; }
68 ._header, ._list { width: <%= app_size %>; }
69 ._list-hover.clone { min-width: <%= app_size %>; }
70 ._notice, ._path, ._resizer { left: <%= app_size %>; }
71</style>
views/index.erb+3−2
@@ -1,5 +1,5 @@
1<!DOCTYPE html>1<!DOCTYPE html>
2<html<%= ' manifest="/manifest.appcache"' if App.production? %> prefix="og: http://ogp.me/ns#" lang="en" class="_booting _theme-<%= app_theme %>">2<html prefix="og: http://ogp.me/ns#" lang="en" class="_booting">
3<head>3<head>
4 <meta charset="utf-8">4 <meta charset="utf-8">
5 <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no,shrink-to-fit=no">5 <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no,shrink-to-fit=no">
@@ -14,6 +14,7 @@ · 8 unmodified lines
14 <meta name="apple-mobile-web-app-capable" content="yes">14 <meta name="apple-mobile-web-app-capable" content="yes">
15 <meta name="apple-mobile-web-app-status-bar-style" content="black">15 <meta name="apple-mobile-web-app-status-bar-style" content="black">
16 <meta name="format-detection" content="telephone=no">16 <meta name="format-detection" content="telephone=no">
17 <meta name="theme-color" content="#eee">
17 <meta name="robots" content="noodp">18 <meta name="robots" content="noodp">
18 <title>DevDocs API Documentation</title>19 <title>DevDocs API Documentation</title>
19 <link rel="canonical" href="<%= canonical_origin %>">20 <link rel="canonical" href="<%= canonical_origin %>">
@@ -32,7 +33,7 @@ · 12 unmodified lines
32 <link rel="mask-icon" href="<%= App.cdn_origin %>/images/webkit-mask-icon.svg" color="#398df0">33 <link rel="mask-icon" href="<%= App.cdn_origin %>/images/webkit-mask-icon.svg" color="#398df0">
33 <%= stylesheet_tag 'application' %>34 <%= stylesheet_tag 'application' %>
34</head>35</head>
35<body<%= %( class="#{app_layout}") if app_layout %>>36<body>
36<noscript class="_fail">DevDocs requires JavaScript to run.</noscript>37<noscript class="_fail">DevDocs requires JavaScript to run.</noscript>
37<%= erb :app -%>38<%= erb :app -%>
38<%= javascript_tag 'application', asset_host: false %>39<%= javascript_tag 'application', asset_host: false %>
views/manifest.erb+0−14
@@ -1,14 +0,0 @@
1CACHE MANIFEST
2# <%= app_theme %> <%= app_size %> <%= app_layout %>
3
4CACHE:
5/
6<%= manifest_asset_urls.join "\n" %>
7<%= doc_index_urls.join "\n" %>
8
9NETWORK:
10/s/
11*
12
13FALLBACK:
14/ /
views/other.erb+3−2
@@ -1,10 +1,11 @@
1<!DOCTYPE html>1<!DOCTYPE html>
2<html lang="en" class="_booting _theme-<%= app_theme %>">2<html lang="en" class="_booting">
3<head>3<head>
4 <meta charset="utf-8">4 <meta charset="utf-8">
5 <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no">5 <meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no">
6 <% if doc_index_page? %><meta name="description" content="<%= @doc['name'] %> <%= @doc['release'] %> API documentation with instant search, offline support, keyboard shortcuts, mobile version, and more."><% else %><meta name="robots" content="noindex"><% end %>6 <% if doc_index_page? %><meta name="description" content="<%= @doc['name'] %> <%= @doc['release'] %> API documentation with instant search, offline support, keyboard shortcuts, mobile version, and more."><% else %><meta name="robots" content="noindex"><% end %>
7 <meta name="format-detection" content="telephone=no">7 <meta name="format-detection" content="telephone=no">
8 <meta name="theme-color" content="#eee">
8 <meta property="og:image" content="<%= App.cdn_origin %>/images/icon-320.png">9 <meta property="og:image" content="<%= App.cdn_origin %>/images/icon-320.png">
9 <title>DevDocs<%= " &mdash; #{@doc['full_name']} documentation" if doc_index_page? %></title>10 <title>DevDocs<%= " &mdash; #{@doc['full_name']} documentation" if doc_index_page? %></title>
10 <link rel="canonical" href="<%= canonical_origin %><%= request.path %>">11 <link rel="canonical" href="<%= canonical_origin %><%= request.path %>">
@@ -13,7 +14,7 @@ · 2 unmodified lines
13 <link rel="search" type="application/opensearchdescription+xml" href="/opensearch.xml" title="Search DevDocs">14 <link rel="search" type="application/opensearchdescription+xml" href="/opensearch.xml" title="Search DevDocs">
14 <%= stylesheet_tag 'application' %>15 <%= stylesheet_tag 'application' %>
15</head>16</head>
16<body<%= %( class="#{app_layout}") if app_layout %> data-doc="<%= CGI::escape_html @doc.to_json %>">17<body data-doc="<%= CGI::escape_html @doc.to_json %>">
17<noscript class="_fail">DevDocs requires JavaScript to run.</noscript>18<noscript class="_fail">DevDocs requires JavaScript to run.</noscript>
18<%= erb :app -%>19<%= erb :app -%>
19<%= javascript_tag 'application', asset_host: false %><% unless App.production? %>20<%= javascript_tag 'application', asset_host: false %><% unless App.production? %>
views/service-worker.js.erb+49−0
@@ -0,0 +1,49 @@
1<%# The name of the cache to store responses in %>
2<%# If the cache name changes DevDocs is assumed to be updated %>
3const cacheName = '<%= service_worker_cache_name %>';
4
5<%# Url's to cache when the service worker is installed %>
6const urlsToCache = [
7 '/',
8 '/favicon.ico',
9 '/manifest.json',
10 '<%= service_worker_asset_urls.join "',\n '" %>',
11 '<%= doc_index_urls.join "',\n '" %>',
12];
13
14<%# Set-up the cache %>
15self.addEventListener('install', event => {
16 self.skipWaiting();
17
18 event.waitUntil(
19 caches.open(cacheName).then(cache => cache.addAll(urlsToCache)),
20 );
21});
22
23<%# Remove old caches %>
24self.addEventListener('activate', event => {
25 event.waitUntil((async () => {
26 const keys = await caches.keys();
27 const jobs = keys.map(key => key !== cacheName ? caches.delete(key) : Promise.resolve());
28 return Promise.all(jobs);
29 })());
30});
31
32<%# Handle HTTP requests %>
33self.addEventListener('fetch', event => {
34 event.respondWith((async () => {
35 const cachedResponse = await caches.match(event.request);
36 if (cachedResponse) return cachedResponse;
37
38 const url = new URL(event.request.url);
39
40 <%# Attempt to return the index page from the cache if the user is visiting a url like devdocs.io/offline or devdocs.io/javascript/global_objects/array/find %>
41 <%# The index page will handle the routing %>
42 if (url.origin === location.origin && !url.pathname.includes('.')) {
43 const cachedIndex = await caches.match('/');
44 if (cachedIndex) return cachedIndex;
45 }
46
47 return fetch(event.request);
48 })());
49});
views/unsupported.erb+3−3
@@ -11,9 +11,9 @@ · 10 unmodified lines
11 <p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:</p>11 <p class="_fail-text">DevDocs is an API documentation browser which supports the following browsers:</p>
12 <ul class="_fail-list">12 <ul class="_fail-list">
13 <li>Recent versions of Firefox, Chrome, or Opera</li>13 <li>Recent versions of Firefox, Chrome, or Opera</li>
14 <li>Safari 9.1+</li>14 <li>Safari 11.1+</li>
15 <li>Edge 16+</li>15 <li>Edge 17+</li>
16 <li>iOS 10+</li>16 <li>iOS 11.3+</li>
17 </ul>17 </ul>
18 <p class="_fail-text">18 <p class="_fail-text">
19 If you're unable to upgrade, we apologize.19 If you're unable to upgrade, we apologize.