History

Add tcllib scraper

Tcllib is effectively Tcl's extended stdlib and so is worth including alongside it. The only issue with this docset is that the `html` module page is broken because they embed unencoded html tags in bad places, causing the page to parse incorrectly.

Chris Roberts committed 1 year ago

Commit 31cf4af1 parent 8b3f5527 files · +105 −-2Browse files

Unified · Split

7 files
  1. lib/docs/filters/tcllib/clean_html.rb +36 −0
  2. lib/docs/filters/tcllib/entries.rb +22 −0
  3. lib/docs/filters/tcllib/nop.rb +9 −0
  4. lib/docs/scrapers/tcllib.rb +39 −0
  5. public/icons/docs/tcllib/16.png +0 −0
  6. public/icons/docs/tcllib/16@2x.png +0 −0
  7. public/icons/docs/tcllib/SOURCE +1 −0
lib/docs/filters/tcllib/clean_html.rb+36−0
@@ -0,0 +1,36 @@
1module Docs
2 class Tcllib
3 class CleanHtmlFilter < Filter
4 def call
5 css("hr").remove()
6 xpath("./div/text()").remove() # Navigation text content e.g. [ | | | ]
7 css("div.markdown > a").remove() # Navigation links
8
9
10 # Fix up ToC links
11 css('a[name]').each do |node|
12 node.parent['id'] = node['name']
13 node.before(node.children).remove unless node['href']
14 end
15
16 # Relies on the above ToC fixup
17 keywords = at_css('#keywords')
18 if !keywords.nil?
19 keywords.next_sibling.remove()
20 keywords.remove()
21 css('a[href="#keywords"]').remove()
22 end
23
24 # Downrank headings for styling
25 css('h2').each do |node|
26 node.name = 'h3'
27 end
28 css('h1').each do |node|
29 node.name = 'h2'
30 end
31
32 doc
33 end
34 end
35 end
36end
lib/docs/filters/tcllib/entries.rb+22−0
@@ -0,0 +1,22 @@
1module Docs
2 class Tcllib
3 class EntriesFilter < Docs::EntriesFilter
4 def get_name
5 # The first word after the `NAME` heading
6 name = at_css('h1 + p')
7 return name.content.strip.split[0]
8 end
9
10 def get_type
11 # The types are the categories as indicated on each page (and on the
12 # root page, toc0.md)
13 category = at_css('a[name="category"]')
14 if !category.nil?
15 return category.parent.next.next.content
16 end
17 return 'Unfiled'
18 end
19 end
20 end
21end
22
lib/docs/filters/tcllib/nop.rb+9−0
@@ -0,0 +1,9 @@
1module Docs
2 class Tcllib
3 class NopFilter < Filter
4 def call
5 doc
6 end
7 end
8 end
9end
lib/docs/scrapers/tcllib.rb+39−0
@@ -0,0 +1,39 @@
1module Docs
2 class Tcllib < UrlScraper
3 self.name = 'Tcllib'
4 self.type = 'simple'
5 self.slug = 'tcllib'
6 self.release = '2.0'
7 self.base_url = 'https://core.tcl-lang.org/tcllib/doc/trunk/embedded/md/'
8 self.root_path = 'toc0.md'
9 self.links = {
10 home: 'https://core.tcl-lang.org/tcllib/doc/trunk/embedded/index.md',
11 code: 'https://sourceforge.net/projects/tcllib/files/tcllib/'
12 }
13
14 html_filters.push 'tcllib/entries', 'tcllib/clean_html', 'title'
15 # The docs have incorrect <base> elements, so we should just skip that
16 html_filters.replace('apply_base_url', 'tcllib/nop')
17
18 options[:root_title] = 'Tcllib Documentation'
19 options[:container] = '.content'
20 options[:skip] = [
21 # Full of broken links, path improperly duplicates "tcllib" segment
22 'tcllib/toc.md',
23 # The other ones aren't terribly useful
24 'toc.md', 'toc1.md', 'toc2.md',
25 # Keyword index
26 'index.md'
27 ]
28
29 options[:attribution] = <<-HTML
30 Licensed under the <a href="https://core.tcl-lang.org/tcllib/doc/trunk/embedded/md/tcllib/files/devdoc/tcllib_license.md">BSD license</a>
31 HTML
32
33
34 def get_latest_version(opts)
35 doc = fetch_doc('https://core.tcl-lang.org/tcllib/doc/trunk/embedded/index.md', opts)
36 doc.at_css('strong').content.scan(/([0-9.]+)/)[0][0]
37 end
38 end
39end
public/icons/docs/tcllib/16.png+0−0

Binary file not shown.

public/icons/docs/tcllib/16@2x.png+0−0

Binary file not shown.

public/icons/docs/tcllib/SOURCE+1−0
@@ -0,0 +1 @@
1https://commons.wikimedia.org/wiki/File:Tcl.svg