BnetScraper is a Nokogiri-based scraper of Battle.net profile information. Currently this only includes Starcraft2.
Run gem install bnet_scraper or add gem 'bnet_scraper' to your Gemfile.
All of the scrapers take an options hash, and can be created by either passing a URL string for the profile URL or passing the account information in the options hash. Thus, either of these two approaches work:
BnetScraper::Starcraft2::ProfileScraper.new(url: 'http://us.battle.net/sc2/en/profile/12345/1/TestAccount/')
BnetScraper::Starcraft2::ProfileScraper.new(bnet_id: '12345', account: 'TestAccount', gateway: 'us')Note that the latter will assume subregion "1", which is North America. Make sure you're also passing the correct subregion along,
if you want to scrape a profile in a different subregion. For a full mapping of regions, see BnetScraper::Starcraft2::REGIONS
There are several scrapers that pull various information. They are:
- BnetScraper::Starcraft2::ProfileScraper - collects basic profile information and an array of league URLs
- BnetScraper::Starcraft2::LeagueScraper - collects data on a particular league for a particular Battle.net account
- BnetScraper::Starcraft2::AchievementScraper - collects achievement data for the account.
- BnetScraper::Starcraft2::MatchHistoryScraper - collects the 25 most recent matches played on the account
All scrapers have a #scrape method that triggers the scraping and storage. By default they will return the result,
but an additional #output method exists to retrieve the results subsequent times without re-scraping.
To clarify some of the confusion that might result from changing terminology and no general consensus, BnetScraper now uses naming relative to the battle.net URL scheme to avoid confusion, especially regarding "subregion" and "gateway":
<gateway>.battle.net/sc2/<locale>/profile/<bnet_id>/<subregion>/<name>
Note that "region" is not in there - it is the freely named key for a combination of gateway and subregion, it does not appear on battle.net anywhere.
This pulls basic profile information for an account, as well as an array of league URLs. This is a good starting point for league scraping as it provides the league URLs necessary to do supplemental scraping.
scraper = BnetScraper::Starcraft2::ProfileScraper.new(url: 'http://us.battle.net/sc2/en/profile/2377239/1/Demon/')
scraper.scrape
# => {
bnet_id: '2377239',
account: 'Demon',
bnet_index: 1,
race: 'Protoss',
wins: '684',
achievement_points: '3630',
current_solo_league: 'Not Yet Ranked',
highest_solo_league: 'Platinum',
current_team_league: 'Not Yet Ranked',
highest_team_league: 'Diamond',
career_games: '1568',
games_this_season: '0',
most_played: '4v4',
leagues: [
{
name: "1v1 Platinum Rank 95",
id: "96905",
href: "http://us.battle.net/sc2/en/profile/2377239/1/Demon/ladder/96905#current-rank"
}
]
}This pulls information on a specific league for a specific account. It is best used either in conjunction with a profile scrape that profiles a URL, or if you happen to know the specific league_id and can pass it as an option.
scraper = BnetScraper::Starcraft2::LeagueScraper.new(league_id: '12345', account: 'Demon', bnet_id: '2377239')
scraper.scrape
# => {
season: '6',
name: 'Aleksander Pepper',
division: 'Diamond',
size: '4v4',
random: false,
bnet_id: '2377239',
account: 'Demon'
}This pulls achievement information for an account. Note that currently only returns the overall achievements, not the in-depth, by-category achievement information.
scraper = BnetScraper::Starcraft2::AchievementScraper.new(url: 'http://us.battle.net/sc2/en/profile/2377239/1/Demon/')
scraper.scrape
# => {
recent: [
{ title: 'Blink of an Eye', description: 'Complete round 24 in "Starcraft Master" without losing any stalkers', earned: '3/5/2012' },
{ title: 'Whack-a-Roach', description: 'Complete round 9 in "Starcraft Master" in under 45 seconds', earned: '3/5/2012' },
{ title: 'Safe Zone', description: 'Complete round 8 in "Starcraft Master" without losing any stalkers', earned: '3/5/2012' },
{ title: 'Starcraft Master', description: 'Complete all 30 rounds in "Starcraft Master"', earned: '3/5/2012' },
{ title: 'Starcraft Expert', description: 'Complete any 25 rounds in "Starcraft Master"', earned: '3/5/2012' },
{ title: 'Starcraft Apprentice', description: 'Complete any 20 rounds in "Starcraft Master"', earned: '3/5/2012' }
],
showcase: [
{ title: 'Hot Shot', description: 'Finish a Qualification Round with an undefeated record.' },
{ title: 'Starcraft Master', description: 'Complete all rounds in "Starcraft Master"' },
{ title: 'Team Protoss 500', description: 'Win 500 team league matches as Protoss' },
{ title: 'Night of the Living III', description: 'Survive 15 Infested Horde Attacks in the "Night 2 Die" mode of the "Left 2 Die" scenario.' },
{ title: 'Team Top 100 Diamond', description: 'Finish a Season in Team Diamond Division' }
],
progress: {
liberty_campaign: '1580',
exploration: '480',
custom_game: '330',
cooperative: '660',
quick_match: '170'
}
}This pulls the 25 most recent matches played for an account. Note that this is only as up-to-date as battle.net is, and will likely not be as fast as in-game.
scraper = BnetScraper::Starcraft2::MatchHistoryScraper.new(url: 'http://us.battle.net/sc2/en/profile/2377239/1/Demon/')
scraper.scrape
# => {
wins: '15',
losses: '10',
matches: [
{ map_name: 'Bx Monobattle - Sand Canyon (Fix)', outcome: :win, type: 'Custom', date: '3/12/2012' },
{ map_name: 'Deadlock Ridge', outcome: :loss, type: '4v4', date: '3/12/2012' },
{ map_name: 'District 10', outcome: :win, type: '4v4', date: '3/12/2012' },
# ...
]
}Scraping is only possible if the site is up. Use this if you want to verify the failed scrape is because the site is down:
BnetScraper::Starcraft2::Status.na # => 'Online'
BnetScraper::Starcraft2::Status.fea # => 'Offline'
BnetScraper::Starcraft2::Status.cn # => nil (China is unsupported)
BnetScraper::Starcraft2::Status.fetch # => [
{:region=>"North America", :status=>"Online"},
{:region=>"Europe", :status=>"Online"},
{:region=>"Korea", :status=>"Online"},
{:region=>"South-East Asia", :status=>"Online"}
]I would love to see contributions! Please send a pull request with a feature branch containing specs (Chances are excellent I will break it if you do not) and I will take a look. Please do not change the version as I tend to bundle multiple fixes together before releasing a new version anyway.
Written by Andrew Nordman, see LICENSE for details.