History

Check for updates based on modified date for scrapers without release

Jasper van Merle committed 7 years ago

Commit cd436321 parent 6ee169320 files · +186 −115Browse files

Unified · Split

20 files
  1. docs/Scraper-Reference.md +1 −1
  2. lib/docs/core/doc.rb +103 −1
  3. lib/docs/core/scraper.rb +0 −85
  4. lib/docs/scrapers/c.rb +8 −0
  5. lib/docs/scrapers/chef.rb +2 −3
  6. lib/docs/scrapers/cpp.rb +9 −0
  7. lib/docs/scrapers/haskell.rb +5 −4
  8. lib/docs/scrapers/http.rb +2 −0
  9. lib/docs/scrapers/markdown.rb +4 −0
  10. lib/docs/scrapers/mdn/css.rb +2 −0
  11. lib/docs/scrapers/mdn/dom.rb +2 −0
  12. lib/docs/scrapers/mdn/dom_events.rb +2 −0
  13. lib/docs/scrapers/mdn/html.rb +2 −0
  14. lib/docs/scrapers/mdn/javascript.rb +2 −0
  15. lib/docs/scrapers/mdn/mdn.rb +6 −0
  16. lib/docs/scrapers/mdn/svg.rb +2 −0
  17. lib/docs/scrapers/mdn/xslt_xpath.rb +2 −0
  18. lib/docs/scrapers/openjdk.rb +1 −1
  19. lib/docs/scrapers/support_tables.rb +7 −0
  20. lib/tasks/updates.thor +24 −20
docs/Scraper-Reference.md+1−1
@@ -187,7 +187,7 @@ More information about how filters work is available on the [Filter Reference](. · 186 unmodified lines
187187
188188 ## Keeping scrapers up-to-date
189189
190−In order to keep scrapers up-to-date the `get_latest_version(options, &block)` method should be overridden by all scrapers that define the `self.release` attribute. This method should return the latest version of the documentation that is being scraped. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.
190+In order to keep scrapers up-to-date the `get_latest_version(options, &block)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.
191191
192192 To make life easier, there are a few utility methods that you can use in `get_latest_version`:
193193 * `fetch(url, options, &block)`
lib/docs/core/doc.rb+103−1
@@ -152,7 +152,6 @@ module Docs · 151 unmodified lines
152152 end
153153 end
154154
155−
156155 def initialize
157156 raise NotImplementedError, "#{self.class} is an abstract class and cannot be instantiated." if self.class.abstract
158157 end
@@ -164,5 +163,108 @@ module Docs · 5 unmodified lines
164163 def build_pages(&block)
165164 raise NotImplementedError
166165 end
166+
167+ def get_scraper_version(opts, &block)
168+ if self.class.method_defined?(:options) and !options[:release].nil?
169+ block.call options[:release]
170+ else
171+ # If options[:release] does not exist, we return the Epoch timestamp of when the doc was last modified in DevDocs production
172+ fetch_json('https://devdocs.io/docs.json', opts) do |json|
173+ items = json.select {|item| item['name'] == self.class.name}
174+ items = items.map {|item| item['mtime']}
175+ block.call items.max
176+ end
177+ end
178+ end
179+
180+ # Should return the latest version of this documentation
181+ # If options[:release] is defined, it should be in the same format
182+ # If options[:release] is not defined, it should return the Epoch timestamp of when the documentation was last updated
183+ # If the docs will never change, simply return '1.0.0'
184+ def get_latest_version(options, &block)
185+ raise NotImplementedError
186+ end
187+
188+ # Returns whether or not this scraper is outdated.
189+ #
190+ # The default implementation assumes the documentation uses a semver(-like) approach when it comes to versions.
191+ # Patch updates are ignored because there are usually little to no documentation changes in bug-fix-only releases.
192+ #
193+ # Scrapers of documentations that do not use this versioning approach should override this method.
194+ #
195+ # Examples of the default implementation:
196+ # 1 -> 2 = outdated
197+ # 1.1 -> 1.2 = outdated
198+ # 1.1.1 -> 1.1.2 = not outdated
199+ def is_outdated(scraper_version, latest_version)
200+ scraper_parts = scraper_version.to_s.split(/\./).map(&:to_i)
201+ latest_parts = latest_version.to_s.split(/\./).map(&:to_i)
202+
203+ # Only check the first two parts, the third part is for patch updates
204+ [0, 1].each do |i|
205+ break if i >= scraper_parts.length or i >= latest_parts.length
206+ return true if latest_parts[i] > scraper_parts[i]
207+ return false if latest_parts[i] < scraper_parts[i]
208+ end
209+
210+ false
211+ end
212+
213+ private
214+
215+ #
216+ # Utility methods for get_latest_version
217+ #
218+
219+ def fetch(url, options, &block)
220+ headers = {}
221+
222+ if options.key?(:github_token) and url.start_with?('https://api.github.com/')
223+ headers['Authorization'] = "token #{options[:github_token]}"
224+ end
225+
226+ options[:logger].debug("Fetching #{url}")
227+
228+ Request.run(url, { headers: headers }) do |response|
229+ if response.success?
230+ block.call response.body
231+ else
232+ options[:logger].error("Couldn't fetch #{url} (response code #{response.code})")
233+ block.call nil
234+ end
235+ end
236+ end
237+
238+ def fetch_doc(url, options, &block)
239+ fetch(url, options) do |body|
240+ block.call Nokogiri::HTML.parse(body, nil, 'UTF-8')
241+ end
242+ end
243+
244+ def fetch_json(url, options, &block)
245+ fetch(url, options) do |body|
246+ block.call JSON.parse(body)
247+ end
248+ end
249+
250+ def get_npm_version(package, options, &block)
251+ fetch_json("https://registry.npmjs.com/#{package}", options) do |json|
252+ block.call json['dist-tags']['latest']
253+ end
254+ end
255+
256+ def get_latest_github_release(owner, repo, options, &block)
257+ fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", options, &block)
258+ end
259+
260+ def get_github_tags(owner, repo, options, &block)
261+ fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", options, &block)
262+ end
263+
264+ def get_github_file_contents(owner, repo, path, options, &block)
265+ fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", options) do |json|
266+ block.call(Base64.decode64(json['content']))
267+ end
268+ end
167269 end
168270 end
lib/docs/core/scraper.rb+0−85
@@ -132,35 +132,6 @@ module Docs · 131 unmodified lines
132132 end
133133 end
134134
135− def get_latest_version(options, &block)
136− raise NotImplementedError
137− end
138−
139− # Returns whether or not this scraper is outdated.
140− #
141− # The default implementation assumes the documentation uses a semver(-like) approach when it comes to versions.
142− # Patch updates are ignored because there are usually little to no documentation changes in bug-fix-only releases.
143− #
144− # Scrapers of documentations that do not use this versioning approach should override this method.
145− #
146− # Examples of the default implementation:
147− # 1 -> 2 = outdated
148− # 1.1 -> 1.2 = outdated
149− # 1.1.1 -> 1.1.2 = not outdated
150− def is_outdated(scraper_version, latest_version)
151− scraper_parts = scraper_version.split(/\./).map(&:to_i)
152− latest_parts = latest_version.split(/\./).map(&:to_i)
153−
154− # Only check the first two parts, the third part is for patch updates
155− [0, 1].each do |i|
156− break if i >= scraper_parts.length or i >= latest_parts.length
157− return true if latest_parts[i] > scraper_parts[i]
158− return false if latest_parts[i] < scraper_parts[i]
159− end
160−
161− false
162− end
163−
164135 private
165136
166137 def request_one(url)
@@ -231,62 +202,6 @@ module Docs · 64 unmodified lines
231202 {}
232203 end
233204
234− #
235− # Utility methods for get_latest_version
236− #
237−
238− def fetch(url, options, &block)
239− headers = {}
240−
241− if options.key?(:github_token) and url.start_with?('https://api.github.com/')
242− headers['Authorization'] = "token #{options[:github_token]}"
243− end
244−
245− options[:logger].debug("Fetching #{url}")
246−
247− Request.run(url, { headers: headers }) do |response|
248− if response.success?
249− block.call response.body
250− else
251− options[:logger].error("Couldn't fetch #{url} (response code #{response.code})")
252− block.call nil
253− end
254− end
255− end
256−
257− def fetch_doc(url, options, &block)
258− fetch(url, options) do |body|
259− block.call Nokogiri::HTML.parse body, nil, 'UTF-8'
260− end
261− end
262−
263− def fetch_json(url, options, &block)
264− fetch(url, options) do |body|
265− json = JSON.parse(body)
266− block.call json
267− end
268− end
269−
270− def get_npm_version(package, options, &block)
271− fetch_json("https://registry.npmjs.com/#{package}", options) do |json|
272− block.call json['dist-tags']['latest']
273− end
274− end
275−
276− def get_latest_github_release(owner, repo, options, &block)
277− fetch_json("https://api.github.com/repos/#{owner}/#{repo}/releases/latest", options, &block)
278− end
279−
280− def get_github_tags(owner, repo, options, &block)
281− fetch_json("https://api.github.com/repos/#{owner}/#{repo}/tags", options, &block)
282− end
283−
284− def get_github_file_contents(owner, repo, path, options, &block)
285− fetch_json("https://api.github.com/repos/#{owner}/#{repo}/contents/#{path}", options) do |json|
286− block.call(Base64.decode64(json['content']))
287− end
288− end
289−
290205 module FixInternalUrlsBehavior
291206 def self.included(base)
292207 base.extend ClassMethods
lib/docs/scrapers/c.rb+8−0
@@ -26,6 +26,14 @@ module Docs · 25 unmodified lines
2626 Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
2727 HTML
2828
29+ def get_latest_version(options, &block)
30+ fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', options) do |doc|
31+ link = doc.at_css('a[title^="File:"]')
32+ date = link.content.scan(/(\d+)\./)[0][0]
33+ block.call DateTime.strptime(date, '%Y%m%d').to_time.to_i
34+ end
35+ end
36+
2937 private
3038
3139 def file_path_for(*)
lib/docs/scrapers/chef.rb+2−3
@@ -49,9 +49,8 @@ module Docs · 48 unmodified lines
4949 end
5050
5151 def get_latest_version(options, &block)
52− fetch_doc('https://docs-archive.chef.io/', options) do |doc|
53− cell = doc.at_css('.main-archives > tr:nth-child(2) > td:nth-child(2)')
54− block.call cell.content.sub(/Chef Client /, '')
52+ fetch_doc('https://downloads.chef.io/chef', options) do |doc|
53+ block.call doc.at_css('h1.product-heading > span').content.strip
5554 end
5655 end
5756 end
lib/docs/scrapers/cpp.rb+9−0
@@ -34,6 +34,15 @@ module Docs · 33 unmodified lines
3434 Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
3535 HTML
3636
37+ # Same as get_latest_version in lib/docs/scrapers/c.rb
38+ def get_latest_version(options, &block)
39+ fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', options) do |doc|
40+ link = doc.at_css('a[title^="File:"]')
41+ date = link.content.scan(/(\d+)\./)[0][0]
42+ block.call DateTime.strptime(date, '%Y%m%d').to_time.to_i
43+ end
44+ end
45+
3746 private
3847
3948 def file_path_for(*)
lib/docs/scrapers/haskell.rb+5−4
@@ -10,7 +10,7 @@ module Docs · 9 unmodified lines
1010
1111 html_filters.push 'haskell/entries', 'haskell/clean_html'
1212
13− options[:container] = ->(filter) { filter.subpath.start_with?('users_guide') ? '.body' : '#content' }
13+ options[:container] = ->(filter) {filter.subpath.start_with?('users_guide') ? '.body' : '#content'}
1414
1515 options[:only_patterns] = [/\Alibraries\//, /\Ausers_guide\//]
1616 options[:skip_patterns] = [
@@ -70,9 +70,10 @@ module Docs · 53 unmodified lines
7070 end
7171
7272 def get_latest_version(options, &block)
73− fetch_doc('https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/', options) do |doc|
74− label = doc.at_css('.related > ul > li:last-child').content
75− block.call label.scan(/([0-9.]+)/)[0][0]
73+ fetch_doc('https://downloads.haskell.org/~ghc/latest/docs/html/', options) do |doc|
74+ links = doc.css('a').to_a
75+ versions = links.map {|link| link['href'].scan(/ghc-([0-9.]+)/)}
76+ block.call versions.find {|version| !version.empty?}[0][0]
7677 end
7778 end
7879 end
lib/docs/scrapers/http.rb+2−0
@@ -7,6 +7,8 @@ module Docs · 6 unmodified lines
77
88 html_filters.push 'http/clean_html', 'http/entries', 'title'
99
10+ options[:mdn_tag] = 'HTTP'
11+
1012 options[:root_title] = 'HTTP'
1113 options[:title] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? false : filter.default_title }
1214 options[:container] = ->(filter) { filter.current_url.host == 'tools.ietf.org' ? '.content' : nil }
lib/docs/scrapers/markdown.rb+4−0
@@ -13,5 +13,9 @@ module Docs · 12 unmodified lines
1313 &copy; 2004 John Gruber<br>
1414 Licensed under the BSD License.
1515 HTML
16+
17+ def get_latest_version(options, &block)
18+ block.call '1.0.0'
19+ end
1620 end
1721 end
lib/docs/scrapers/mdn/css.rb+2−0
@@ -6,6 +6,8 @@ module Docs · 5 unmodified lines
66
77 html_filters.push 'css/clean_html', 'css/entries', 'title'
88
9+ options[:mdn_tag] = 'CSS'
10+
911 options[:root_title] = 'CSS'
1012
1113 options[:skip] = %w(/CSS3 /Media/Visual /paged_media /Media/TV /Media/Tactile)
lib/docs/scrapers/mdn/dom.rb+2−0
@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
88
99 html_filters.push 'dom/clean_html', 'dom/entries', 'title'
1010
11+ options[:mdn_tag] = 'XSLT_Reference'
12+
1113 options[:root_title] = 'DOM'
1214
1315 options[:skip] = %w(
lib/docs/scrapers/mdn/dom_events.rb+2−0
@@ -9,6 +9,8 @@ module Docs · 8 unmodified lines
99 html_filters.insert_after 'clean_html', 'dom_events/clean_html'
1010 html_filters.push 'dom_events/entries', 'title'
1111
12+ options[:mdn_tag] = 'events'
13+
1214 options[:root_title] = 'DOM Events'
1315
1416 options[:skip] = %w(/MozOrientation)
lib/docs/scrapers/mdn/html.rb+2−0
@@ -7,6 +7,8 @@ module Docs · 6 unmodified lines
77
88 html_filters.push 'html/clean_html', 'html/entries', 'title'
99
10+ options[:mdn_tag] = 'HTML'
11+
1012 options[:root_title] = 'HTML'
1113
1214 options[:title] = ->(filter) do
lib/docs/scrapers/mdn/javascript.rb+2−0
@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
88
99 html_filters.push 'javascript/clean_html', 'javascript/entries', 'title'
1010
11+ options[:mdn_tag] = 'JavaScript'
12+
1113 options[:root_title] = 'JavaScript'
1214
1315 # Don't want
lib/docs/scrapers/mdn/mdn.rb+6−0
@@ -21,6 +21,12 @@ module Docs · 20 unmodified lines
2121 Licensed under the Creative Commons Attribution-ShareAlike License v2.5 or later.
2222 HTML
2323
24+ def get_latest_version(opts, &block)
25+ fetch_json("https://developer.mozilla.org/en-US/docs/feeds/json/tag/#{options[:mdn_tag]}", opts) do |json|
26+ block.call DateTime.parse(json[0]['pubdate']).to_time.to_i
27+ end
28+ end
29+
2430 private
2531
2632 def process_response?(response)
lib/docs/scrapers/mdn/svg.rb+2−0
@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
88
99 html_filters.push 'svg/clean_html', 'svg/entries', 'title'
1010
11+ options[:mdn_tag] = 'XSLT_Reference'
12+
1113 options[:root_title] = 'SVG'
1214
1315 options[:title] = ->(filter) do
lib/docs/scrapers/mdn/xslt_xpath.rb+2−0
@@ -8,6 +8,8 @@ module Docs · 7 unmodified lines
88
99 html_filters.push 'xslt_xpath/clean_html', 'xslt_xpath/entries', 'title'
1010
11+ options[:mdn_tag] = 'XSLT_Reference'
12+
1113 options[:root_title] = 'XSLT'
1214
1315 options[:only_patterns] = [/\A\/XSLT/, /\A\/XPath/]
lib/docs/scrapers/openjdk.rb+1−1
@@ -105,7 +105,7 @@ module Docs · 104 unmodified lines
105105 end
106106 end
107107
108− block.call latest_version.to_s
108+ block.call latest_version
109109 end
110110 end
111111 end
lib/docs/scrapers/support_tables.rb+7−0
@@ -178,5 +178,12 @@ module Docs · 177 unmodified lines
178178 </p>
179179 </div>
180180 HTML
181+
182+ def get_latest_version(options, &block)
183+ fetch('https://feeds.feedburner.com/WhenCanIUse?format=xml', options) do |body|
184+ timestamp = body.scan(/<updated>([^<]+)<\/updated>/)[0][0]
185+ block.call DateTime.parse(timestamp).to_time.to_i
186+ end
187+ end
181188 end
182189 end
lib/tasks/updates.thor+24−20
@@ -1,7 +1,7 @@
11 class UpdatesCLI < Thor
22 # The GitHub user that is allowed to upload reports
33 # TODO: Update this before creating a PR
4− UPLOAD_USER = 'jmerle'
4+ UPLOAD_USER = 'devdocs-bot'
55
66 # The repository to create an issue in when uploading the results
77 # TODO: Update this before creating a PR
@@ -59,25 +59,19 @@ class UpdatesCLI < Thor · 51 unmodified lines
5959 private
6060
6161 def check_doc(doc, opts)
62− # Newer scraper versions always come before older scraper versions
63− # Therefore, the first item's release value is the latest scraper version
64− #
65− # For example, a scraper could scrape 3 versions: 10, 11 and 12
66− # doc.versions.first would be the scraper for version 12
67− instance = doc.versions.first.new
68−
69− scraper_version = instance.class.method_defined?(:options) ? instance.options[:release] : nil
70− return error_result(doc, '`options[:release]` does not exist') if scraper_version.nil?
71−
7262 logger.debug("Checking #{doc.name}")
7363
74− instance.get_latest_version(opts) do |latest_version|
75− return {
76− name: doc.name,
77− scraper_version: scraper_version,
78− latest_version: latest_version,
79− is_outdated: instance.is_outdated(scraper_version, latest_version)
80− }
64+ instance = doc.versions.first.new
65+
66+ instance.get_scraper_version(opts) do |scraper_version|
67+ instance.get_latest_version(opts) do |latest_version|
68+ return {
69+ name: doc.name,
70+ scraper_version: format_version(scraper_version),
71+ latest_version: format_version(latest_version),
72+ is_outdated: instance.is_outdated(scraper_version, latest_version)
73+ }
74+ end
8175 end
8276 rescue NotImplementedError
8377 logger.warn("Couldn't check #{doc.name}, get_latest_version is not implemented")
@@ -87,6 +81,15 @@ class UpdatesCLI < Thor · 3 unmodified lines
8781 raise
8882 end
8983
84+ def format_version(version)
85+ str = version.to_s
86+
87+ # If the version is numeric and greater than or equal to 1e9 it's probably a timestamp
88+ return str if str.match(/^(\d)+$/).nil? or str.to_i < 1e9
89+
90+ DateTime.strptime(str, '%s').strftime('%B %-d, %Y')
91+ end
92+
9093 def error_result(doc, reason)
9194 {
9295 name: doc.name,
@@ -199,14 +202,15 @@ class UpdatesCLI < Thor · 106 unmodified lines
199202 ]
200203
201204 results_str = results.select {|result| !result.nil?}.join("\n\n")
205+ travis_str = ENV['TRAVIS'].nil? ? '' : "\n\nThis issue was created by Travis CI build [##{ENV['TRAVIS_BUILD_NUMBER']}](#{ENV['TRAVIS_BUILD_WEB_URL']})."
202206
203− title = "Documentation versions report for #{Date.today.strftime('%B')} 2019"
207+ title = "Documentation versions report for #{Date.today.strftime('%B %Y')}"
204208 body = <<-MARKDOWN
205209 ## What is this?
206210
207211 This is an automatically created issue which contains information about the version status of the documentations available on DevDocs. The results of this report can be used by maintainers when updating outdated documentations.
208212
209−Maintainers can close this issue when all documentations are up-to-date. This issue is automatically closed when the next report is created.
213+Maintainers can close this issue when all documentations are up-to-date. This issue is automatically closed when the next report is created.#{travis_str}
210214
211215 ## Results
212216

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t