History

salt_stack: finish scraper and filters

Jasper van Merle committed 7 years ago

Commit f3d5a6e1 parent 96a72024 files · +51 −11Browse files

Unified · Split

4 files
  1. assets/javascripts/templates/pages/about_tmpl.coffee +6 −0
  2. lib/docs/filters/salt_stack/clean_html.rb +11 −0
  3. lib/docs/filters/salt_stack/entries.rb +12 −1
  4. lib/docs/scrapers/salt_stack.rb +22 −10
assets/javascripts/templates/pages/about_tmpl.coffee+6−0
@@ -638,6 +638,12 @@ credits = [ · 637 unmodified lines
638638 'MIT',
639639 'https://raw.githubusercontent.com/rust-lang/rust/master/LICENSE-MIT'
640640 ], [
641+ 'Salt Stack',
642+ '2019 SaltStack',
643+ 'Apache',
644+ 'https://raw.githubusercontent.com/saltstack/salt/develop/LICENSE'
645+ ],
646+ [
641647 'Sass',
642648 '2006-2016 Hampton Catlin, Nathan Weizenbaum, and Chris Eppstein',
643649 'MIT',
lib/docs/filters/salt_stack/clean_html.rb+11−0
@@ -4,6 +4,17 @@ module Docs · 3 unmodified lines
44 def call
55 css('.headerlink').remove
66
7+ css('div[class^="highlight-"]').each do |node|
8+ node.name = 'pre'
9+ node['data-language'] = node['class'].scan(/highlight-([a-z]+)/i)[0][0]
10+ node.content = node.content.strip
11+ end
12+
13+ css('.function > dt').each do |node|
14+ node.name = 'h3'
15+ node.content = node.content
16+ end
17+
718 doc
819 end
920 end
lib/docs/filters/salt_stack/entries.rb+12−1
@@ -16,12 +16,23 @@ module Docs · 15 unmodified lines
1616 end
1717
1818 def get_type
19− slug.split('/', 2).first
19+ slug.split('/', 3)[1]
2020 end
2121
2222 def include_default_entry?
2323 slug.split('/').last.start_with? 'salt'
2424 end
25+
26+ def additional_entries
27+ entries = []
28+
29+ css('.function > h3').each do |node|
30+ name = node.content.remove('salt.').split('(')[0] + '()'
31+ entries << [name, node['id']]
32+ end
33+
34+ entries
35+ end
2536 end
2637 end
2738 end
lib/docs/scrapers/salt_stack.rb+22−10
@@ -1,8 +1,19 @@
11 module Docs
2− class SaltStack < UrlScraper
3− self.type = 'salt_stack'
4− self.release = '2018.3.2'
5− self.base_url = 'https://docs.saltstack.com/en/latest/ref/'
2+ # The official documentation website is heavily rate-limited
3+ #
4+ # The documentation can be generated like this (replace 2019.2 with the correct tag):
5+ # $ git clone https://github.com/saltstack/salt.git --branch 2019.2 --depth 1
6+ # $ cd salt/doc
7+ # $ pip install sphinx
8+ # $ make html
9+ #
10+ # The generated html can be found in salt/doc/_build/html
11+ class SaltStack < FileScraper
12+ self.type = 'simple'
13+ self.slug = 'salt_stack'
14+ self.release = '2019.2.0'
15+ self.base_url = 'https://docs.saltstack.com/en/latest/'
16+ self.root_path = 'ref/index.html'
617 self.links = {
718 home: 'https://www.saltstack.com/',
819 code: 'https://github.com/saltstack/salt'
@@ -10,15 +21,16 @@ module Docs · 1 unmodified line
1021
1122 html_filters.push 'salt_stack/clean_html', 'salt_stack/entries'
1223
13− options[:only_patterns] = [
14− %r{^[^/]+/all/}
15− ]
16−
17− options[:container] = '.body-content'
24+ options[:only_patterns] = [/all\//]
25+ options[:container] = '.body-content > .section'
1826
1927 options[:attribution] = <<-HTML
20− &copy; 2018 SaltStack.<br>
28+ &copy; 2019 SaltStack.<br>
2129 Licensed under the Apache License, Version 2.0.
2230 HTML
31+
32+ def get_latest_version(opts)
33+ get_latest_github_release('saltstack', 'salt', opts)
34+ end
2335 end
2436 end

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t