History

migrate cpp scraper from filescraper to urlscraper

Enoc committed 5 years ago

Commit e17bc841 parent a754d509 files · +189 −171Browse files

Showing the changes to lib/docs/scrapers/cppref (2 of 9 files). Show all

Unified · Split

2 files
  1. lib/docs/scrapers/cppref/cpp.rb +28 −0
  2. lib/docs/scrapers/cppref/cppref.rb +29 −0
lib/docs/scrapers/cppref/cpp.rb+28−0
@@ -0,0 +1,28 @@
1+module Docs
2+ class Cpp < Cppref
3+ self.name = 'C++'
4+ self.slug = 'cpp'
5+ self.type = 'c'
6+ self.base_url = 'https://en.cppreference.com/w/cpp/'
7+
8+ html_filters.insert_before 'cppref/clean_html', 'cpp/entries'
9+
10+ options[:root_title] = 'C++ Programming Language'
11+
12+ options[:skip] = %w(
13+ language/extending_std.html
14+ language/history.html
15+ regex/ecmascript.html
16+ regex/regex_token_iterator/operator_cmp.html
17+ )
18+
19+ # Same as get_latest_version in lib/docs/scrapers/c.rb
20+ def get_latest_version(opts)
21+ doc = fetch_doc('https://en.cppreference.com/w/Cppreference:Archives', opts)
22+ link = doc.at_css('a[title^="File:"]')
23+ date = link.content.scan(/(\d+)\./)[0][0]
24+ DateTime.strptime(date, '%Y%m%d').to_time.to_i
25+ end
26+
27+ end
28+end
lib/docs/scrapers/cppref/cppref.rb+29−0
@@ -0,0 +1,29 @@
1+module Docs
2+ class Cppref < UrlScraper
3+ self.abstract = true
4+ self.type = 'cppref'
5+ self.root_path = 'header'
6+
7+ html_filters.insert_before 'clean_html', 'cppref/fix_code'
8+ html_filters.push 'cppref/clean_html', 'title'
9+ # 'cpp20/entries',
10+ options[:decode_and_clean_paths] = true
11+ options[:container] = '#content'
12+ options[:title] = false
13+ options[:skip] = %w(language/history.html)
14+
15+ options[:skip_patterns] = [
16+ /experimental/
17+ ]
18+
19+ options[:attribution] = <<-HTML
20+ &copy; cppreference.com<br>
21+ Licensed under the Creative Commons Attribution-ShareAlike Unported License v3.0.
22+ HTML
23+
24+ # def get_latest_version
25+
26+ # end
27+
28+ end
29+end

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t