History

Implement working crawling and section building for salt

Boris Bera committed 8 years ago

Commit b7075dd1 parent 9d60cc83 files · +25 −3Browse files

Unified · Split

3 files
  1. lib/docs/filters/salt_stack/clean_html.rb +2 −0
  2. lib/docs/filters/salt_stack/entries.rb +17 −2
  3. lib/docs/scrapers/salt_stack.rb +6 −1
lib/docs/filters/salt_stack/clean_html.rb+2−0
@@ -2,6 +2,8 @@ module Docs · 1 unmodified line
2 class SaltStack2 class SaltStack
3 class CleanHtmlFilter < Filter3 class CleanHtmlFilter < Filter
4 def call4 def call
5 css('.headerlink').remove
6
5 doc7 doc
6 end8 end
7 end9 end
lib/docs/filters/salt_stack/entries.rb+17−2
@@ -1,12 +1,27 @@
1module Docs1module Docs
2 class SaltStack2 class SaltStack
3 class EntriesFilter < Docs::EntriesFilter3 class EntriesFilter < Docs::EntriesFilter
4 SALT_REF_RGX = /salt\.([^\.]+)\.([^\s]+)/
5
4 def get_name6 def get_name
5 at_css('h1').content7 header = at_css('h1').content
8
9 ref_match = SALT_REF_RGX.match(header)
10 if ref_match
11 ns, mod = ref_match.captures
12 "#{ns}.#{mod}"
13 else
14 header
15 end
6 end16 end
7 17
8 def get_type18 def get_type
9 'TODO'19 type, _ = slug.split('/', 2)
20 type
21 end
22
23 def include_default_entry?
24 !subpath.end_with?('index.html')
10 end25 end
11 end26 end
12 end27 end
lib/docs/scrapers/salt_stack.rb+6−1
@@ -1,9 +1,14 @@
1module Docs1module Docs
2 class SaltStack < UrlScraper2 class SaltStack < UrlScraper
3 self.type = 'salt_stack'
3 self.release = '2018.3.2'4 self.release = '2018.3.2'
4 self.base_url = 'https://docs.saltstack.com/en/latest/ref/'5 self.base_url = 'https://docs.saltstack.com/en/latest/ref/'
5 6
6 html_filters.push 'salt_stack/entries', 'salt_stack/clean_html'7 html_filters.push 'salt_stack/clean_html', 'salt_stack/entries'
8
9 options[:only_patterns] = [
10 %r{[^/]+/all/}
11 ]
7 12
8 options[:container] = '.body-content'13 options[:container] = '.body-content'
9 14

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t