Skip to content

update for post url change #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 3, 2018
Merged

update for post url change #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
update for post url change
  • Loading branch information
@ccbrown
ccbrown committed Sep 3, 2018
commit b6820517d40ab5e03cf2eac69d0b7519410209c3
8 changes: 3 additions & 5 deletions server/forum_indexer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ func (indexer *ForumIndexer) requestDocument(host, resource string) (*goquery.Do
return goquery.NewDocumentFromReader(resp.Body)
}

var postURLExpression = regexp.MustCompile("^/forum/view-thread/([0-9]+)/page/([0-9]+)#p([0-9]+)")
var postURLExpression = regexp.MustCompile("^/forum/view-post/([0-9]+)")
var threadURLExpression = regexp.MustCompile("^/forum/view-thread/([0-9]+)")
var forumURLExpression = regexp.MustCompile("^/forum/view-forum/([0-9]+)")

Expand Down Expand Up @@ -152,12 +152,10 @@ func ScrapeForumPosts(doc *goquery.Document, locale *Locale, timezone *time.Loca
href := sel.AttrOr("href", "")
if match := postURLExpression.FindStringSubmatch(href); match != nil {
n, _ := strconv.Atoi(match[1])
post.ThreadId = n
n, _ = strconv.Atoi(match[2])
post.PageNumber = n
n, _ = strconv.Atoi(match[3])
post.Id = n
} else if match := threadURLExpression.FindStringSubmatch(href); match != nil {
n, _ := strconv.Atoi(match[1])
post.ThreadId = n
post.ThreadTitle = sel.Text()
} else if match := forumURLExpression.FindStringSubmatch(href); match != nil {
n, _ := strconv.Atoi(match[1])
Expand Down
1 change: 0 additions & 1 deletion server/forum_indexer_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ func TestScrapeForumPosts(t *testing.T) {
assert.Equal(t, "Chris", p.Poster)
assert.Equal(t, "Photos of the Fan Meetup", p.ThreadTitle)
assert.Equal(t, "Announcements", p.ForumName)
assert.Equal(t, 1, p.PageNumber)
assert.Equal(t, "we had a great ti<strong>m</strong>e too!", p.BodyHTML)
assert.Equal(t, int64(1486332365), p.Time.Unix())
}
Expand Down
1 change: 0 additions & 1 deletion server/forum_post.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ type ForumPost struct {
Poster string `json:"poster"`
ThreadId int `json:"thread_id"`
ThreadTitle string `json:"thread_title"`
PageNumber int `json:"page_number"`
ForumId int `json:"forum_id"`
ForumName string `json:"forum_name"`
}
Expand Down
4 changes: 2 additions & 2 deletions server/testdata/forum-posts.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,12 @@
<div class="layoutBox1 layoutBoxFull defaultTheme">
<div class="topBar first last"><div class="breadcrumb"><a href="/account/view-profile/Chris">View Profile</a><span class="separator">»</span>Recent posts by Chris</div></div>
<div class="layoutBoxContent"><table class="forumTable forumPostListTable"><tr><td class="content-container"><div class="content">we had a great ti<strong>m</strong>e too!</div><div class="signature">Lead Developer. Follow us on: <a href="http://www.twitter.com/pathofexile">Twitter</a> | <a href="http://www.youtube.com/grindinggear">YouTube</a> | <a href="http://www.facebook.com/pathofexile">Facebook</a> | Contact <a href="http://www.pathofexile.com/support">Support</a> if you need help!<br />
<img src="https://p7p4m6s5.ssl.hwcdn.net/public/chris/maps3.png" alt="maps3" /></div></td><td class="post_info"><div><div class="centered"><a class="bright bold medium" href="/forum/view-thread/1830139/page/1#p14168107">View post</a></div> <div class="avatar frame1">
<img src="https://p7p4m6s5.ssl.hwcdn.net/public/chris/maps3.png" alt="maps3" /></div></td><td class="post_info"><div><div class="centered"><a class="bright bold medium" href="/forum/view-post/14168107">View post</a></div> <div class="avatar frame1">
<img src="https://web.poecdn.com/gen/image/YTozOntpOjA7aTowO2k6/MjthOjI6e3M6MjoiaWQi/O2k6Mjc3O3M6NDoic2l6/ZSI7czo2OiJhdmF0YXIi/O31pOjE7aToxO30,/f28981be51/Path_of_Exile_Gallery_Image.jpg" alt="Avatar" />
</div><div class="posted-by"><a class="posted-by-link" href="#p14168107">Posted by</a> <span class="profile-link staff post_by_account" ><a href="/account/view-profile/Chris"><img class="achievement" src="https://web.poecdn.com/image/icons/achievements/7.png?v=11" alt="Completed 7 Challenges" title="Completed 7 Challenges" />Chris</a></span><br>on <span class="post_date">Feb 5, 2017, 2:06:05 PM</span><div class="roleLabel staffText">Grinding Gear Games</div></div><div><span class="bright">Forum:</span> <a href="/forum/view-forum/54">Announcements</a></div><div><span class="bright">Thread:</span> <a href="/forum/view-thread/1830139">Photos of the Fan Meetup</a></div></div></td></tr><tr class="even"><td class="content-container"><div class="content">Yesterday, we attended the San Francisco fan meetup and had a great time! It was our busiest one yet, and we were able to stay for three hours answering questions and meeting fans. Today's news post has some photos of the event! <a href="http://pathofexile.com/forum/view-thread/1830139">Read More</a>.

</div><div class="signature">Lead Developer. Follow us on: <a href="http://www.twitter.com/pathofexile">Twitter</a> | <a href="http://www.youtube.com/grindinggear">YouTube</a> | <a href="http://www.facebook.com/pathofexile">Facebook</a> | Contact <a href="http://www.pathofexile.com/support">Support</a> if you need help!<br />
<img src="https://p7p4m6s5.ssl.hwcdn.net/public/chris/maps3.png" alt="maps3" /></div><div class="last_edited_by">Last edited by Chris on Feb 6, 2017 8:31:55 AM</div></td><td class="post_info"><div><div class="centered"><a class="bright bold medium" href="/forum/view-thread/1830139/page/1#p14168056">View post</a></div> <div class="avatar frame1">
<img src="https://p7p4m6s5.ssl.hwcdn.net/public/chris/maps3.png" alt="maps3" /></div><div class="last_edited_by">Last edited by Chris on Feb 6, 2017 8:31:55 AM</div></td><td class="post_info"><div><div class="centered"><a class="bright bold medium" href="/forum/view-post/p14168056">View post</a></div> <div class="avatar frame1">
<img src="https://web.poecdn.com/gen/image/YTozOntpOjA7aTowO2k6/MjthOjI6e3M6MjoiaWQi/O2k6Mjc3O3M6NDoic2l6/ZSI7czo2OiJhdmF0YXIi/O31pOjE7aToxO30,/f28981be51/Path_of_Exile_Gallery_Image.jpg" alt="Avatar" />
</div><div class="posted-by"><a class="posted-by-link" href="#p14168056">Posted by</a> <span class="profile-link staff post_by_account" ><a href="/account/view-profile/Chris"><img class="achievement" src="https://web.poecdn.com/image/icons/achievements/7.png?v=11" alt="Completed 7 Challenges" title="Completed 7 Challenges" />Chris</a></span><br>on <span class="post_date">Feb 5, 2017, 1:48:29 PM</span><div class="roleLabel staffText">Grinding Gear Games</div></div><div><span class="bright">Forum:</span> <a href="/forum/view-forum/54">Announcements</a></div><div><span class="bright">Thread:</span> <a href="/forum/view-thread/1830139">Photos of the Fan Meetup</a></div></div></td></tr><tr><td class="content-container"><div class="content">Chris, Jonathan and Nick are going to be in San Francisco next week to hold a press tour for various upcoming announcements. If you're interested in meeting with them, they'll be hosting a fan meet-up on Saturday February 4th at 4pm (PST). If you're in or around the area, you should come on by! <a href="http://pathofexile.com/forum/view-thread/1827130">Read More</a>.

Expand Down