History

migrate cpp scraper from filescraper to urlscraper

Enoc committed 5 years ago

Commit e17bc841 parent a754d509 files · +189 −171Browse files

Showing the changes to lib/docs/scrapers/cppref (2 of 9 files). Show all

Unified · Split

2 files
  1. lib/docs/scrapers/cppref/cpp.rb +28 −0
  2. lib/docs/scrapers/cppref/cppref.rb +29 −0
lib/docs/scrapers/cppref/cpp.rb+28−0
@@ -0,0 +1,28 @@
1module Docs
2 class Cpp < Cppref
3 self.name = 'C++'
4 self.slug = 'cpp'
5 self.type = 'c'
6 self.base_url = 'https://en.cppreference.com/w/cpp/'
7
8 html_filters.insert_before 'cppref/clean_html', 'cpp/entries'
9
10 options[:root_title] = 'C++ Programming Language'
11
12 options[:skip] = %w(
13 language/extending_std.html
14 language/history.html
15 regex/ecmascript.html
16 regex/regex_token_iterator/operator_cmp.html
17 )
18
19 # Same as get_latest_version in lib/docs/scrapers/c.rb
20 def get_latest_version(opts)
21 doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts)
22 link = doc.at_css('a[title^="File:"]')
23 date = link.content.scan(/(\d+)\./)[0][0]
24 DateTime.strptime(date, '%Y%m%d').to_time.to_i
25 end
26
27 end
28end
lib/docs/scrapers/cppref/cppref.rb+29−0
@@ -0,0 +1,29 @@
1module Docs
2 class Cppref < UrlScraper
3 self.abstract = true
4 self.type = 'cppref'
5 self.root_path = 'header'
6
7 html_filters.insert_before 'clean_html', 'cppref/fix_code'
8 html_filters.push 'cppref/clean_html', 'title'
9 # 'cpp20/entries',
10 options[:decode_and_clean_paths] = true
11 options[:container] = '#content'
12 options[:title] = false
13 options[:skip] = %w(language/history.html)
14
15 options[:skip_patterns] = [
16 /experimental/
17 ]
18
19 options[:attribution] = <<-HTML
20 &copy; cppreference.com<br>
21 Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
22 HTML
23
24 # def get_latest_version
25
26 # end
27
28 end
29end

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t