Merged in changes which make CDN serving of static files easy, adds bower management of external static files, and transitions the datatracker to use django's staticfiles framework.

 - Legacy-Id: 9976

Author: levkowetz

.gitignore

@@ -30,4 +30,5 @@
 /lib
 /share
 /include
+/static
 /latest-coverage.json

README-CDN.rst

@@ -0,0 +1,183 @@
+================================================================================
+		  Serving Static Datatracker Files via a CDN
+================================================================================
+
+Intro
+=====
+
+With release 6.4.0, the way that the static files used by the datatracker are
+handled changes substantially.  Static files were previously versioned under a
+top-level ``static/`` directory, but this is not the case any more.  External
+files (such as for instance ``jquery.min.js``) are now placed under
+``ietf/externals/static/`` and updated using a tool called bower_, while
+datatracker-specific files (images, css, js, etc.) are located under
+``ietf/static/ietf/`` and ``ietf/secr/static/secr/`` respectively.
+
+The following sections provide more details about handling of internals,
+externals, and how deployment is done.
+
+
+Serving Static Files via CDN
+============================
+
+Production Mode
+---------------
+
+If resources served over a CDN and/or with a high max-age don't have different
+URLs for different versions, then any component upgrade which is accompanied
+by a change in template functionality will have a long transition time
+during which the new pages are served with old components, with possible
+breakage.  We want to avoid this.
+
+The intention is that after a release has been checked out, but before it is
+deployed, the standard django 'collectstatic' management command will be
+run, resulting in all static files being collected from their working
+directory location and placed in an appropiate location for serving via CDN.
+This location will have the datatracker release version as part of its URL,
+so that after the deployment of a new release, the CDN will be forced to fetch
+the appropriate static files for that release.
+
+An important part of this is to set up the ``STATIC_ROOT`` and ``STATIC_URL``
+settings appropriately.  In 6.4.0, the setting is as follows in production
+mode::
+
+    STATIC_URL = "https://www.ietf.org/lib/dt/%s/"%__version__
+    STATIC_ROOT = CDN_ROOT + "/a/www/www6s/lib/dt/%s/"%__version__
+
+The result is that all static files collected via the ``collectstatic``
+command will be placed in a location served via CDN, with the release
+version being part of the URL.
+
+Development Mode
+----------------
+
+In development mode, ``STATIC_URL`` is set to ``/static/``, and Django's
+``staticfiles`` infrastructure makes the static files available under that
+local URL root (unless you set
+``settings.SERVE_CDN_FILES_LOCALLY_IN_DEV_MODE`` to ``False``).  It is not
+necessary to actually populate the ``static/`` directory by running
+``collectstatic`` in order for static files to be served when running
+``ietf/manage.py runserver`` -- the ``runserver`` command has extra support
+for finding and serving static files without running collectstatic.
+
+In order to work backwards from a file served in development mode to the
+location from which it is served, the mapping is as follows::
+
+	==============================	==============================	
+	Development URL			Working copy location
+	==============================	==============================	
+	localhost:8000/static/ietf/*	ietf/static/ietf/*
+	localhost:8000/static/secr/*	ietf/secr/static/secr/*
+	localhost:8000/static/*		ietf/externals/static/*
+	==============================	==============================	
+
+Handling of External Javascript and CSS Components 
+==================================================
+
+In order to make it easy to keep track of and upgrade external components,
+these are now handled by a tool called ``bower``, via a new management
+command ``bower_install``.  Each external component is listed in a file
+``ietf/bower.json``.  In order to update the version of a component listed in
+``ietf/bower.json``, or add a new one, you should edit ``bower.json``, and
+then run the management command::
+
+    $ ietf/manage.py bower_install
+
+(Not surprisingly, you need to have bower_ installed in order to use this
+management command.)
+
+That command will fetch the required version of each external component listed
+in ``bower.json`` (actually, it will do this for *all* ``bower.json`` files
+found in the ``static/`` directories of all ``INSTALLED_APPS`` and the
+directories in ``settings.STATICFILES_DIRS``), saving them temporarily under
+``.tmp/bower_components/``; it will then extract the relevant production
+``js`` and ``css`` files and place them in an appropriately named directory
+under ``ietf/externals/static/``.  The latter location is taken from
+``COMPONENT_ROOT`` in ``settings.py``.
+
+Managing external components via bower has the additional benefit of
+managing dependencies -- components that have dependencies will pull in
+these, so that they also are placed under ``ietf/externals/static/``.
+You still have to manually add the necessary stylesheet and/or javascript
+references to your templates, though.
+
+The ``bower_install`` command is not run automatically by ``bin/mkrelease``,
+since it needs an updated ``bower.json`` in order to do anything interesting.
+So when you're intending to update an external web asset to a newer version,
+you need to edit the ``bower.json`` file, run ``manage.py bower_install``,
+verify that the new version doesn't break things, and then commit the new
+files under ``ietf/externals/static/`` and the updated ``bower.json``.
+
+.. _bower: http://bower.io/
+
+The  ``ietf/externals/static/`` Directory
+-----------------------------------------
+
+The directory ``ietf/externals/static/`` holds a number of subdirectories
+which hold distribution files for external client-side components, collected
+by ``bower_install`` as described above.  Currently
+(01 Aug 2015) this means ``js`` and ``css`` components and fonts.
+
+These components each reside in their own subdirectory, which is named with
+the component name::
+
+    henrik@zinfandel $ ls -l ietf/externals/static/
+    total 40
+    drwxr-xr-x 6 henrik henrik 4096 Jul 25 15:25 bootstrap
+    drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 bootstrap-datepicker
+    drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 font-awesome
+    drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery
+    drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery.cookie
+    drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptmono
+    drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptsans
+    drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptserif
+    drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2
+    drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2-bootstrap-css
+
+The ``pt*`` fonts are an exception, in that there is no bower component
+available for these fonts, so they have been put in place manually.
+
+
+Handling of Internal Static Files
+=================================
+
+Previous to this release, internal static files were located under
+``static/``, mixed together with the external components.  They are now
+located under ``ietf/static/ietf/`` and ``ietf/secr/static/secr``, and will be
+collected for serving via CDN by the ``collectstatic`` command.  Any static
+files associated with a particular app will be handled the same way (which
+means that all ``admin/`` static files automatically will be handled correctly, too).
+
+Handling of Customised Bootstrap Files
+======================================
+
+We are using a customised version of Bootstrap_, which is handled specially,
+by a SVN externals definition in ``ietf/static/ietf``.  That pulls the content
+of the ``bootstrap/dist/`` directory (which is generated by running ``grunt``
+in the ``bootstrap/`` directory) into ``ietf/static/ietf/bootstrap``, from
+where it is collected by ``collectstatic``.
+
+Changes to Template Files
+=========================
+
+In order to make the template files refer to the correct versioned CDN URL
+(as given by the STATIC_URL root) all references to static files in the
+templates have been updated to use the ``static`` template tag when referring
+to static files.  This will automatically result in both serving static files
+from the right place in development mode, and referring to the correct
+versioned URL in production mode and the simpler ``/static/`` urls in
+development mode.
+
+.. _bootstrap: http://getbootstrap.com/
+
+Deployment
+==========
+
+During deployment, it is now necessary to run the management command::
+
+  $ ietf/manage.py collectstatic
+
+before activating a new release.  
+
+The deployment ``README`` file at ``/a/www/ietf-datatracker/README`` has been
+updated accordingly.

bin/mkrelease

@@ -166,6 +166,10 @@ DEV="$(printf %d.%d.%d.dev0 $MAJOR $MINOR $NEXT)"
 
 #cd $DIR ??
 
+note "Collecting static files"
+$do ietf/manage.py collectstatic --noinput --ignore=bower.json --ignore='README.*'
+$do svn commit static/lib/ -m "Updated static files under static/lib/"
+
 # note "Checking that there's a recent test-crawler log"
 # touch -d $RDATE .svn/.latest-commit
 # TCLOG=$(ls -t ../test-crawl-*.log | head -n 1)

bin/test-crawl

@@ -27,9 +27,13 @@ parser.add_argument('--validator-nu', dest='validator_nu', action='store_true',
                     help='Use validator.nu instead of html5lib for HTML validation')
 parser.add_argument('--pedantic', action='store_true',
                     help='Stop the crawl on the first HTML validation issue')
+parser.add_argument('--random', action='store_true',
+                    help='Crawl URLs randomly')
 parser.add_argument('--validate-all', dest='validate_all', action='store_true', default=False,
                     help='Run html 5 validation on all pages, without skipping similar urls. '
                     '(The default is to only run validation on one of /foo/1/, /foo/2/, /foo/3/, etc.)')
+parser.add_argument('-v', '--verbose', action='store_true', default=False,
+                    help='Be more verbose')
 
 args = parser.parse_args()
 
@@ -59,13 +63,18 @@ MAX_URL_LENGTH = 500
 
 # --- Functions ---
 
+def note(s):
+    if args.verbose:
+        sys.stderr.write(s)
+        sys.stderr.write('\n')
+
 def strip_url(url):
     if url.startswith("http://testserver"):
         url = url[len("http://testserver"):]
     return url
 
 def extract_html_urls(content):
-    for m in re.finditer(r'(<(?:a|link) [^>]*href=[\'"]([^"]+)[\'"][^>]*>)', content):
+    for m in re.finditer(r'(<(?:(?:a|link) [^>]*href|(?:img|script) [^>]*src)=[\'"]([^"]+)[\'"][^>]*>)', content):
         if re.search(r'rel=["\']?nofollow["\']', m.group(1)):
             continue
 
@@ -108,20 +117,36 @@ def check_html_valid(url, response, args):
     key = url
     if not args.validate_all:
         # derive a key for urls like this by replacing primary keys
-        key = re.sub("/[0-9.]+/",   "/nnnn/", key)
-        key = re.sub("/.+@.+/",    "/x@x.org/", key)
-        key = re.sub("#.*$",       "", key)
         key = re.sub("\?.*$",       "", key)
+        key = re.sub("#.*$",       "", key)
+        key = re.sub("/.+@.+/",    "/x@x.org/", key)
+        key = re.sub("/[0-9.]+/",   "/nnnn/", key)
+        key = re.sub("/[0-9.]+/",   "/mmmm/", key)
+        key = re.sub("/ag/[a-z0-9-]+/",   "/ag/foo/", key)
+        key = re.sub("/area/[a-z0-9-]+/",   "/area/foo/", key)
+        key = re.sub("/bcp[0-9]+/",   "/bcpnnn/", key)
+        key = re.sub("/conflict-review-[a-z0-9-]+/",   "/conflrev-foo/", key)
+        key = re.sub("/dir/[a-z0-9-]+/",   "/dir/foo/", key)
+        key = re.sub("/draft-[a-z0-9-]+/",   "/draft-foo/", key)
+        key = re.sub("/group/[a-z0-9-]+/",   "/group/foo/", key)
+        key = re.sub("/ipr/search/.*",   "/ipr/search/", key)
+        key = re.sub("/release/[0-9dev.]+/",   "/release/n.n.n/", key)
         key = re.sub("/rfc[0-9]+/",   "/rfcnnnn/", key)
-        key = re.sub("/wg/[a-z0-9-]+/",   "/wg/foo/", key)
         key = re.sub("/rg/[a-z0-9-]+/",   "/rg/foo/", key)
-        key = re.sub("/ipr/[0-9]+/",   "/ipr/nnnn/", key)
-        key = re.sub("/draft-[a-z0-9-]+/",   "/draft-foo/", key)
+        key = re.sub("/secr/srec/nnnn/[0-9a-z-]+/", "/secr/sreq/nn/bar/", key)
+        key = re.sub("/state/[a-z0-9-]+/",   "/state/foo/", key)
+        key = re.sub("/state/[a-z0-9-]+/[a-z0-9-]+/",   "/state/foo/bar/", key)
+        key = re.sub("/status-change-[a-z0-9-]+/",   "/statchg-foo/", key)
+        key = re.sub("/std[0-9]+/",   "/stdnnn/", key)
+        key = re.sub("/submit/status/nnnn/[0-9a-f]+/", "/submit/status/nnnn/bar/", key)
+        key = re.sub("/team/[a-z0-9-]+/",   "/team/foo/", key)
+        key = re.sub("/wg/[a-z0-9-]+/",   "/wg/foo/", key)
+
         for slug in doc_types:
             key = re.sub("/%s-.*/"%slug, "/%s-nnnn/"%slug, key)
 
     if not key in validated_urls:
-
+        note('Validate: %-32s: %s' % (url[:32], key))
         # These URLs have known issues, skip them until those are fixed
         if re.search('(/secr|admin/)|/doc/.*/edit/info/', url):
             log("%s blacklisted; skipping HTML validation" % url)
@@ -159,6 +184,14 @@ def check_html_valid(url, response, args):
                                 (pos, code))
                 warnings += 1
 
+def skip_url(url):
+    for pattern in (
+            "^/community/[0-9]+/remove_document/",
+            "^/community/personal/",
+        ):
+        if re.search(pattern, url):
+            return True
+    return False
 
 def log(s):
     print(s)
@@ -237,12 +270,18 @@ if __name__ == "__main__":
             sys.exit(1)
 
     while urls:
-        # popitem() is documented to be random, but really isn't
-        url = random.choice(urls.keys())
-        referrer = urls.pop(url)
+        if args.random:
+            # popitem() is documented to be random, but really isn't
+            url = random.choice(urls.keys())
+            referrer = urls.pop(url)
+        else:
+            url, referrer = urls.popitem()
 
         visited.add(url)
 
+        if skip_url(url):
+            continue
+
         try:
             timestamp = datetime.datetime.now()
             r = client.get(url, secure=True, follow=True)
@@ -298,7 +337,7 @@ if __name__ == "__main__":
                         log("=============")
 
             else:
-                tags.append(u"FAIL for %s\n   (from %s)" % (url, referrer))
+                tags.append(u"FAIL  (from %s)" % (referrer, ))
                 errors += 1
 
             if elapsed.total_seconds() > slow_threshold: