Hacker News - 3 Stage Import

Import planning and overview

Planning Stage

This will be a 3 stage import. All should be linked to the main item, 2 and 3 should be linked to 1. 

1. List from Hacker News [ hacker_news ]
2. Details from links linked to main item [ hacker_news_items ]
3. Comments on each item [ hacker_news_comments ]

Stage 1

titles and links from here. https://news.ycombinator.com/ 

• title
• link_url
• score_points
• user_name
• user_url
• comments_number
• comments_list
• comments_processed
• link_processed

Stage 2

import article from link_url in stage 1 and insert into stage 1 table, pass content via readability for formatting. 

Stage 3

Optional comments import per item