refactored away from 'region' toward gateway and subregion identification to mimic battle.net more closely

Author: fx

README.md

@@ -13,9 +13,12 @@ passing the account information in the options hash.  Thus, either of these two
 
 ``` ruby
 BnetScraper::Starcraft2::ProfileScraper.new(url: 'http://us.battle.net/sc2/en/profile/12345/1/TestAccount/')
-BnetScraper::Starcraft2::ProfileScraper.new(bnet_id: '12345', account: 'TestAccount', region: 'na')
+BnetScraper::Starcraft2::ProfileScraper.new(bnet_id: '12345', account: 'TestAccount', gateway: 'us')
 ```
 
+Note that the latter will assume subregion "1", which is North America. Make sure you're also passing the correct subregion along,
+if you want to scrape a profile in a different subregion. For a full mapping of regions, see `BnetScraper::Starcraft2::REGIONS`
+
 There are several scrapers that pull various information.  They are:
 
 * BnetScraper::Starcraft2::ProfileScraper - collects basic profile information and an array of league URLs
@@ -26,6 +29,14 @@ There are several scrapers that pull various information.  They are:
 All scrapers have a `#scrape` method that triggers the scraping and storage.  By default they will return the result,
 but an additional `#output` method exists to retrieve the results subsequent times without re-scraping.
 
+## Terminology
+
+To clarify some of the confusion that might result from changing terminology and no general consensus, BnetScraper now uses naming relative to the battle.net URL scheme to avoid confusion, especially regarding "subregion" and "gateway":
+
+`<gateway>.battle.net/sc2/<locale>/profile/<bnet_id>/<subregion>/<name>`
+
+Note that "region" is not in there - it is the freely named key for a combination of gateway and subregion, it does not appear on battle.net anywhere.
+
 ## BnetScraper::Starcraft2::ProfileScraper
 
 This pulls basic profile information for an account, as well as an array of league URLs.  This is a good starting

lib/bnet_scraper/starcraft2.rb

@@ -11,20 +11,28 @@ module BnetScraper
   # for more details
   module Starcraft2
     REGIONS = {
-      'na'  => { domain: 'us.battle.net', dir: 'en', label: 'North America' },
-      'eu'  => { domain: 'eu.battle.net', dir: 'en', label: 'Europe' },
-      'cn'  => { domain: 'www.battlenet.com.cn', dir: 'zh', label: 'China' },
-      'sea' => { domain: 'sea.battle.net', dir: 'en', label: 'South-East Asia' },
-      'fea' => { domain: 'tw.battle.net', dir: 'zh', label: 'Korea' }
+      'na'  => { gateway: 'us',  subregion: 1, locale: 'en', label: 'North America' },
+      'la'  => { gateway: 'us',  subregion: 2, locale: 'en', label: 'Latin America' },
+      'eu'  => { gateway: 'eu',  subregion: 1, locale: 'en', label: 'Europe' },
+      'ru'  => { gateway: 'eu',  subregion: 2, locale: 'en', label: 'Russia' },
+      'cn'  => { gateway: 'cn',  subregion: 1, locale: 'zh', label: 'China' },
+      'sea' => { gateway: 'sea', subregion: 1, locale: 'en', label: 'South-East Asia' },
+      'tw'  => { gateway: 'tw',  subregion: 1, locale: 'zh', label: 'Taiwan' },
+      'kr'  => { gateway: 'kr',  subregion: 1, locale: 'ko', label: 'Korea' },
+      # There's a subregion 2 for korea and I honestly don't know what the name of that would be
+      'kr2' => { gateway: 'kr',  subregion: 2, locale: 'ko', label: 'Also Korea' }
     }
 
-    REGION_DOMAINS = {
-      'us.battle.net' => 'na',
+    # Gateway Hostname Mapping
+    # This exists primarily because china does things differently from everyone else.
+    HOSTNAMES = {
+      'us.battle.net' => 'us',
       'eu.battle.net' => 'eu',
       'www.battlenet.com.cn' => 'cn',
+      'cn.battle.net' => 'cn',
       'sea.battle.net' => 'sea',
-      'kr.battle.net' => 'fea',
-      'tw.battle.net' => 'fea'
+      'kr.battle.net' => 'kr',
+      'tw.battle.net' => 'tw'
     }
 
     # The armory uses spritemaps that are sequentially named and have a fixed
@@ -88,12 +96,12 @@ module Starcraft2
     # the parameters being sent to `#full_profile_scrape`.
     #
     # @param bnet_id - Battle.net Account ID 
-    # @param account - Battle.net Account Name
-    # @param region  - Battle.net Account Region
+    # @param name    - Battle.net Account Name
+    # @param gateway - Battle.net Account Gateway
     # @return profile_data - Hash containing complete profile and league data
     #   scraped from the website
-    def self.full_profile_scrape bnet_id, account, region = 'na'
-      profile_scraper = ProfileScraper.new bnet_id: bnet_id, account: account, region: region
+    def self.full_profile_scrape bnet_id, name, gateway = 'us'
+      profile_scraper = ProfileScraper.new bnet_id: bnet_id, name: name, gateway: gateway
       profile_output  = profile_scraper.scrape
 
       parsed_leagues = []

lib/bnet_scraper/starcraft2/base_scraper.rb

@@ -1,66 +1,54 @@
 module BnetScraper
   module Starcraft2
     # BaseScraper handles the account information extraction. Each scraper can either be passed a profile URL or
-    # the minimum information needed to access an account.  This means passing in account and bnet_id at minimum.
+    # the minimum information needed to access an account: bnet_id and name. The gateway will default to 'us'
+    # and region will default to the first region for the given gateway.
+    #
     # Both of the following are valid ways to instantiate a scraper for the same account:
     #
     #   BnetScraper::Starcraft2::BaseScraper.new(url: 'http://us.battle.net/sc2/en/profile/12345/1/TestAccount/')
-    #   BnetScraper::Starcraft2::BaseScraper.new(bnet_id: '12345', account: 'TestAccount')
+    #   BnetScraper::Starcraft2::BaseScraper.new(bnet_id: '12345', name: 'TestAccount')
     #
     # The URL scheme is the following:
-    #
-    #   http://<REGION_DOMAIN>/sc2/<REGION_LANG>/profile/<BNET_ID>/<BNET_INDEX>/<ACCOUNT>/
-    #
-    # Note that by default, the region will be set to 'na' if you opt not to specify the URL or region.  The 
-    # scraper uses the short-codes for regions. See `BnetScraper::Starcraft2::REGIONS` for the address 
-    # translations.
+    #   http://<gateway>.battle.net/sc2/<locale>/profile/<bnet_id>/<region>/<name>/
     class BaseScraper
-      attr_reader :bnet_id, :account, :region, :bnet_index, :url
+      attr_reader :bnet_id, :name, :gateway, :subregion, :region, :url
 
       def initialize options = {}
         if options[:url]
           extracted_data = options[:url].match(/http:\/\/(.+)\/sc2\/(.+)\/profile\/(.+)\/(\d{1})\/(.[^\/]+)\//)
           if extracted_data
-            @region     = REGION_DOMAINS[extracted_data[1]]
+            @hostname   = extracted_data[1]
+            @gateway    = HOSTNAMES[@hostname]
+            @locale     = extracted_data[2]
             @bnet_id    = extracted_data[3]
-            @bnet_index = extracted_data[4]
-            @account    = extracted_data[5]
+            @subregion  = extracted_data[4].to_i
+            @name       = extracted_data[5]
             @url        = options[:url]
           else
             raise BnetScraper::InvalidProfileError, "URL provided does not match Battle.net format"
           end
-        elsif options[:bnet_id] && options[:account]
+        elsif options[:bnet_id] && options[:name]
           @bnet_id  = options[:bnet_id]
-          @account  = options[:account]
-          @region   = options[:region] || 'na'
-          if options[:bnet_index]
-            @bnet_index = options[:bnet_index]
-          else
-            set_bnet_index
-          end
-        end
-      end
-
-      # set_bnet_index
-      # 
-      # Because profile URLs have to have a specific bnet_index that is seemingly incalculable,
-      # we must ping both variants to determine the correct bnet_index. We then store that value.
-      def set_bnet_index
-        [1,2].each do |idx|
-          res = Net::HTTP.get_response URI profile_url idx 
-          if res.is_a? Net::HTTPSuccess
-            @bnet_index = idx
-            return
-          end
+          @name     = options[:name]
+          @gateway  = options[:gateway] || 'us'
+          @subregion= (options[:subregion] || 1).to_i
+        else
+          raise BnetScraper::InvalidProfileError, "Required options missing"
         end
+        
+        @region = REGIONS.find{|region, data| data[:gateway] == @gateway && data[:subregion] == @subregion}
+        raise BnetScraper::InvalidProfileError, "Could not identify region from gateway '#{@gateway}' and subregion '#{@subregion}'" if !@region
+        @region = @region[0]
       end
 
-      def profile_url bnet_index = @bnet_index
-        "http://#{region_info[:domain]}/sc2/#{region_info[:dir]}/profile/#{bnet_id}/#{bnet_index}/#{account}/"
+      # Returns the profile URL generated from associated data
+      # Note: while China has it's own hostname/domain, cn.battle.net will
+      # redirect to it properly.
+      def profile_url
+        "http://#{gateway}.battle.net/sc2/#{region_info[:locale]}/profile/#{bnet_id}/#{region_info[:subregion]}/#{name}/"
       end
 
-      # converts region short-code to region-based URL information
-      #   'na' => { domain: 'us.battle.net', :dir: 'en' }
       def region_info
         REGIONS[region]
       end

lib/bnet_scraper/starcraft2/league_scraper.rb

@@ -3,19 +3,19 @@ module Starcraft2
     # This pulls information on a specific league for a specific account.  It is best used either in conjunction with a
     # profile scrape that profiles a URL, or if you happen to know the specific league\_id and can pass it as an option.
     #
-    #  scraper = BnetScraper::Starcraft2::LeagueScraper.new(league_id: '12345', account: 'Demon', bnet_id: '2377239')
+    #  scraper = BnetScraper::Starcraft2::LeagueScraper.new(league_id: '12345', name: 'Demon', bnet_id: '2377239')
     #  scraper.scrape
     #  # => {
     #    season: '6',
-    #    name: 'Aleksander Pepper',
-    #    division: 'Diamond',
+    #    division: 'Aleksander Pepper',
+    #    league: 'Diamond',
     #    size: '4v4',
     #    random: false,
     #    bnet_id: '2377239',
-    #    account: 'Demon'
+    #    name: 'Demon'
     #  }
     class LeagueScraper < BaseScraper
-      attr_reader :league_id, :season, :size, :random, :name, :division
+      attr_reader :league_id, :season, :size, :random, :league, :division
 
       # @param [String] url - The league URL on battle.net
       # @return [Hash] league_data - Hash of data extracted
@@ -37,7 +37,7 @@ def scrape
           header_regex = /Season (\d{1}) - \s+(\dv\d)( Random)? (\w+)\s+Division (.+)/
           header_values = value.match(header_regex).to_a
           header_values.shift()
-          @season, @size, @random, @division, @name = header_values
+          @season, @size, @random, @league, @division = header_values
 
           @random = !@random.nil?
           output
@@ -50,11 +50,11 @@ def output
         {
           season: @season,
           size: @size,
-          name: @name,
           division: @division,
+          league: @league,
           random: @random,
           bnet_id: @bnet_id,
-          account: @account
+          name: @name
         }
       end
     end

lib/bnet_scraper/starcraft2/profile_scraper.rb

@@ -7,8 +7,8 @@ module Starcraft2
     #   scraper.scrape
     #   # => {
     #     bnet_id: '2377239',
-    #     account: 'Demon',
-    #     bnet_index: 1,
+    #     name: 'Demon',
+    #     subregion: 1,
     #     race: 'Protoss',
     #     wins: '684',
     #     achievement_points: '3630',
@@ -117,8 +117,8 @@ def get_league_list
       def output
          {
           bnet_id: @bnet_id,
-          account: @account,
-          bnet_index: @bnet_index,
+          name: @name,
+          subregion: @subregion,
           race: @race,
           current_solo_league: @current_solo_league,
           highest_solo_league: @highest_solo_league,

lib/bnet_scraper/starcraft2/status_scraper.rb

@@ -5,7 +5,7 @@ module Starcraft2
     #
     # Examples:
     #   BnetScraper::Starcraft2::Status.na  => 'Online'
-    #   BnetScraper::Starcraft2::Status.fea => 'Offline'
+    #   BnetScraper::Starcraft2::Status.kr => 'Offline'
     #   BnetScraper::Starcraft2::Status.cn  => nil (China is unsupported)
     #   BnetScraper::Starcraft2::Status.fetch => [
     #     {:region=>"North America", :status=>"Online"},{:region=>"Europe", :status=>"Online"},

spec/starcraft2/base_scraper_spec.rb

@@ -14,7 +14,7 @@
 
   describe '#scrape' do
     it 'should raise an error calling scrape' do
-      expect { subject.scrape }.to raise_error NotImplementedError
+      expect { subject.scrape }.to raise_error BnetScraper::InvalidProfileError
     end
   end
 end

spec/starcraft2/league_scraper_spec.rb

@@ -23,15 +23,15 @@
     end
 
     it 'should set the name' do
-      subject.name.should be_nil 
+      subject.division.should be_nil 
       subject.scrape
-      subject.name.should == 'Aleksander Pepper'
+      subject.division.should == 'Aleksander Pepper'
     end
 
     it 'should set the divison' do
-      subject.division.should be_nil
+      subject.league.should be_nil
       subject.scrape
-      subject.division.should == 'Diamond'
+      subject.league.should == 'Diamond'
     end
 
     it 'should set the size' do
@@ -62,12 +62,12 @@
     it 'should return a hash of league data' do
       expected = {
         season: '6',
-        name: 'Aleksander Pepper',
-        division: 'Diamond',
+        division: 'Aleksander Pepper',
+        league: 'Diamond',
         size: '4v4',
         random: false,
         bnet_id: '2377239',
-        account: 'Demon'
+        name: 'Demon'
       }
 
       subject.scrape

spec/starcraft2/profile_scraper_spec.rb

@@ -6,7 +6,7 @@
     let(:subject) { scraper_class.new(url: 'http://us.battle.net/sc2/en/profile/2377239/1/Demon/') }
   end
 
-  subject { BnetScraper::Starcraft2::ProfileScraper.new(bnet_id: '2377239', account: 'Demon') }
+  subject { BnetScraper::Starcraft2::ProfileScraper.new(bnet_id: '2377239', name: 'Demon') }
 
   describe '#get_profile_data' do
     before do
@@ -69,7 +69,7 @@
       expect { scraper.scrape }.to raise_error(BnetScraper::InvalidProfileError)
     end
 
-    context 'account that has not laddered' do
+    context 'name that has not laddered' do
       let(:scraper) {BnetScraper::Starcraft2::ProfileScraper.new(url: 'http://us.battle.net/sc2/en/profile/3354437/1/ClarkeKent/') }
       before do
         scraper.scrape
@@ -85,8 +85,8 @@
     it 'should extract profile data from the response' do
       expected = {
         bnet_id: '2377239',
-        account: 'Demon',
-        bnet_index: 1,
+        name: 'Demon',
+        subregion: 1,
         race: 'Protoss',
         career_games: '1568',
         games_this_season: '0',
@@ -163,8 +163,8 @@
 
       subject.output.should == { 
         bnet_id: '2377239', 
-        account: 'Demon',
-        bnet_index: 1,
+        name: 'Demon',
+        subregion: 1,
         race: nil,
         career_games: nil,
         games_this_season: nil,

spec/starcraft2/status_scraper_spec.rb

@@ -6,7 +6,7 @@
       BnetScraper::Starcraft2::Status.na.should == 'Online'
       BnetScraper::Starcraft2::Status.eu.should == 'Online'
       BnetScraper::Starcraft2::Status.sea.should == 'Online'
-      BnetScraper::Starcraft2::Status.fea.should == 'Online'
+      BnetScraper::Starcraft2::Status.kr.should == 'Online'
     end
   end