From 95ec443ac048f0785ac144f5d8050446476617f0 Mon Sep 17 00:00:00 2001 From: Jeremy Davis Date: Sun, 18 Sep 2016 12:48:58 +1000 Subject: [PATCH 1/5] include info in readme --- README.rst | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 147 insertions(+), 6 deletions(-) diff --git a/README.rst b/README.rst index 485c411..9e52ddf 100644 --- a/README.rst +++ b/README.rst @@ -2,6 +2,11 @@ TurnKey Dev Tracker =================== +.. contents:: Contents: + +Overview +-------- + TurnKey uses GitHub's project management features to track development. Resources: @@ -20,6 +25,16 @@ This tracker is **NOT** intended to be used as: Reporting a bug, issue or feature request ----------------------------------------- +Before creating a new issue on the `Issue Tracker`_, please check to see +if a similar issue already exists. Please read `How to read the Issue +Tracker`_ section below for details. Understanding our `Issue +Tracker Workflow`_ might also be useful. + +If an issue exists already please post a comment showing it also affects +you. Knowing an issue effects multiple users is useful when we decide +how to prioritize limited development resources. Please try and include +any additional information you think might help. + The ideal bug/issue includes: 1) A detailed description of the issue. @@ -34,17 +49,137 @@ The ideal feature request includes: 2) One or more use cases for the feature. 3) Any extra information that might be relevant. -Before creating a new issue on the `Issue Tracker`_, please check to see if a similar issue -already exists. If it does, post a comment showing it also affects you. -Knowing an issue effects multiple users is useful when we decide how -to prioritize limited development resources. Please try and include any additional -information you think might help us close the issue. +Issue Tracker Workflow +---------------------- + +1) Issue reported on the `Issue Tracker`_ (by you!?) +2) Issue triaged by TurnKey team member: + + - Labelled_ with the appropriate tags. Each issue can have multiple + tags, e.g.: + + - *bug* [red] or *feature* (request) [green]; + - *security* / *critical* (when relevant) [deep red]; + - appliance or software the issue relates to [orange] e.g. *lamp*; + - appliance build type the issue relates to (when relevant) [ + blue/green] e.g. *ova (buildtype)*; + - other tags as relevant e.g. *workaround* if a workaround is + documented, *upstream* if the bug is inherited from elsewhere, + etc. + + - Pinned to a milestone_. Each issue can only be pinned to a single + milestone. I.e.: + + - for appliances, generally the next release e.g. *14.2*; + - for other software, the next release that we plan e.g. *tklbam + v1.5*; + - sometimes one of the more vague milestones: *on ice* or *soon + (hopefully)*; + - occasionally none (if we haven't yet decided). + +3) A bugfix or feature is developed (by you!? or us) and a PR (pull + request) is issued against the relevant repo here on GitHub. +4) Preliminary code review occurs. As need be, considerations are + discussed within the PR. +5) The fix or feature is merged into master branch of relevant repo. +6) Issue is closed. +7) If the bug is considered *critical* or *security* then the appliance + is re-released (and the bug is re-pinned to the pervious milestone). +8) Rinse and repeat until ready for next version release... +9) Once the new version is released, the milestone is closed and a new + (next version) milestone is created. Generally a new version is not + released until all the milestone bugs are closed. Any unaddressed + feature requests and new bugs will be pinned to the new milestone. + +How to read the Issue Tracker +----------------------------- + +If you have encountered an issue; either a bug or something that could +be much better (i.e. a feature request), have a look on the +`Issue Tracker`_. + +By default, the Issue Tracker will display all open issues. The +"Filter" search box by default will include ``is:issue is:open``. +Remove ``is:open`` and include the name of the appliance &/or any +other terms you think might be relevant. Hit enter and all the +relevant issues (both open and closed) should be displayed. You may +wish to revise your search to be more or less specific depending on +the amount of results. + +Between the title and the tags (especially the orange ones) you +should be able to see whether your issue has been reported already. +Please note that issues relevant to all appliances will usually +only be tagged *core*; issues relating to a subset of appliances +will often only be tagged with the base appliance name. E.g. an +Apache config bug that effects all LAMP based appliances may only +be tagged *lamp*. + +If the issue you're experiencing has already been reported, you can +further refine your understanding by considering the milestone it's +pinned to and it's state (*open* or *closed*). + +**Milestone**: + +- Current version (e.g. currently *14.1*) + + - if pinned to the current version milestone then the issue should + already be addressed in the current release. If not then + there are 2 possibilities: + + 1) We have re-released the appliance (to include the fix). This + should be resolved if you retry with the current appliance. + 2) We've made a mistake somewhere along the line. If you think that + this has happened, please let us know. + +- Future version (e.g. current open milestone is *14.2*) + + - if pinned to the next version milestone then the issue hasn't + yet been resolved for the current release. To understand the current + status of an issue pinned to a future release milestone, you will + need to also consider it's state (see below). + +**State** (*open* or *closed*): + +- Open: + + - the issue has not yet been addressed. The issue thread may include + additional info about a workaround or other ways to mitigate the + issue. + +- Closed: + + - the issue has been resolved and code has been merged. However the + current release may not yet include the fix/update. You will need to + consider the milestone to be sure. + +When will a bugfixed appliance be released? +------------------------------------------- + +Our `Issue Tracker Workflow`_ means that issues are usually closed +long before we release a new version. So it is expected that the +current appliance may still include some of the known bugs. + +Ideally we'd like to re-release an appliance as soon as we have resolved +a bug. However due to limited resources and the re-release overhead, we +prioritize rebuilds on the basis of bug severity. If a bug is critical +or has serious security implications we will re-release ASAP. If the +basic functionality of the appliance is not directly effected then we +will wait for the next release. + +Obviously this decision making process is subjective. If you think that +we've made a mistake, please comment on the issue and explain your +perspective and we may reconsider. + +If you wish, you can `rebuild the appliance yourself`_, using TKLDev_ +and the latest buildcode. `Buildtasks`_ supports building appliances to +any of our target `build types`_. Contributing as a developer --------------------------- TurnKey is 100% open source and the code for all components is right here on GitHub. -Developers with good ideas are strongly encouraged to be bold and contribute code. Use the source Luke! +Developers with good ideas are strongly encouraged to be bold and contribute code. +Use the source Luke! See the `guidelines and walk through`_. @@ -52,5 +187,11 @@ See the `guidelines and walk through`_. .. _Wiki: https://github.com/turnkeylinux/tracker/wiki/ .. _support forum: http://www.turnkeylinux.org/forum/support/ .. _general discussion forum: http://www.turnkeylinux.org/forum/general/ +.. _Labelled: https://github.com/turnkeylinux/tracker/labels +.. _milestone: https://github.com/turnkeylinux/tracker/milestones +.. _rebuild the appliance yourself: https://www.turnkeylinux.org/docs/howto-build-isos-with-tkldev +.. _TKLDev: https://www.turnkeylinux.org/tkldev +.. _Buildtasks: https://github.com/turnkeylinux/buildtasks +.. _build types: https://www.turnkeylinux.org/docs/builds .. _guidelines and walk through: https://github.com/turnkeylinux/tracker/blob/master/GITFLOW.rst From 2d0d07e1e4d5e7a82b0f6bff4bcce7b41de40feb Mon Sep 17 00:00:00 2001 From: Jeremy Davis Date: Mon, 19 Dec 2016 14:40:29 +1100 Subject: [PATCH 2/5] updated .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index c4ffbba..5f0a32e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ wiki issues +contrib/config.json From 4074709b585e1da4961a264aea48ee7f0d2a98a5 Mon Sep 17 00:00:00 2001 From: Jeremy Davis Date: Mon, 19 Dec 2016 14:41:20 +1100 Subject: [PATCH 3/5] add get-apps.py script & conf --- contrib/apps.json | 710 ++++++++++++++++++++++++++++++++++++ contrib/config.json.example | 3 + contrib/get-apps.py | 46 +++ 3 files changed, 759 insertions(+) create mode 100644 contrib/apps.json create mode 100644 contrib/config.json.example create mode 100755 contrib/get-apps.py diff --git a/contrib/apps.json b/contrib/apps.json new file mode 100644 index 0000000..6af7e37 --- /dev/null +++ b/contrib/apps.json @@ -0,0 +1,710 @@ +[ + { + "release": "14.1", + "app_name": "ansible", + "gh_page": "https://github.com/turnkeylinux-apps/ansible", + "app_page": "https://www.turnkeylinux.org/ansible" + }, + { + "release": "13.0", + "app_name": "appengine-go", + "gh_page": "https://github.com/turnkeylinux-apps/appengine-go", + "app_page": "http://www.turnkeylinux.org/appengine-go" + }, + { + "release": "13.0", + "app_name": "appengine-java", + "gh_page": "https://github.com/turnkeylinux-apps/appengine-java", + "app_page": "http://www.turnkeylinux.org/appengine-java" + }, + { + "release": "13.0", + "app_name": "appengine-python", + "gh_page": "https://github.com/turnkeylinux-apps/appengine-python", + "app_page": "http://www.turnkeylinux.org/appengine-python" + }, + { + "release": "0", + "app_name": "appflower", + "gh_page": "https://github.com/turnkeylinux-apps/appflower", + "app_page": "http://www.turnkeylinux.org/appflower" + }, + { + "release": "14.1", + "app_name": "asp-net-apache", + "gh_page": "https://github.com/turnkeylinux-apps/asp-net-apache", + "app_page": "http://www.turnkeylinux.org/asp-net-apache" + }, + { + "release": "14.1", + "app_name": "b2evolution", + "gh_page": "https://github.com/turnkeylinux-apps/b2evolution", + "app_page": "http://www.turnkeylinux.org/b2evolution" + }, + { + "release": "13.0", + "app_name": "bambooinvoice", + "gh_page": "https://github.com/turnkeylinux-apps/bambooinvoice", + "app_page": "http://www.turnkeylinux.org/bambooinvoice" + }, + { + "release": "0", + "app_name": "bitkey", + "gh_page": "https://github.com/turnkeylinux-apps/bitkey", + "app_page": "http://bitkey.io" + }, + { + "release": "14.1", + "app_name": "bugzilla", + "gh_page": "https://github.com/turnkeylinux-apps/bugzilla", + "app_page": "http://www.turnkeylinux.org/bugzilla" + }, + { + "release": "14.1", + "app_name": "cakephp", + "gh_page": "https://github.com/turnkeylinux-apps/cakephp", + "app_page": "http://www.turnkeylinux.org/cakephp" + }, + { + "release": "14.1", + "app_name": "canvas", + "gh_page": "https://github.com/turnkeylinux-apps/canvas", + "app_page": "http://www.turnkeylinux.org/canvas" + }, + { + "release": "13.0", + "app_name": "clipbucket", + "gh_page": "https://github.com/turnkeylinux-apps/clipbucket", + "app_page": "http://www.turnkeylinux.org/clipbucket" + }, + { + "release": "14.1", + "app_name": "codeigniter", + "gh_page": "https://github.com/turnkeylinux-apps/codeigniter", + "app_page": "http://www.turnkeylinux.org/codeigniter" + }, + { + "release": "14.1", + "app_name": "collabtive", + "gh_page": "https://github.com/turnkeylinux-apps/collabtive", + "app_page": "http://www.turnkeylinux.org/collabtive" + }, + { + "release": "14.1", + "app_name": "concrete5", + "gh_page": "https://github.com/turnkeylinux-apps/concrete5", + "app_page": "http://www.turnkeylinux.org/concrete5" + }, + { + "release": "14.1", + "app_name": "core", + "gh_page": "https://github.com/turnkeylinux-apps/core", + "app_page": "https://www.turnkeylinux.org/core" + }, + { + "release": "14.1", + "app_name": "couchdb", + "gh_page": "https://github.com/turnkeylinux-apps/couchdb", + "app_page": "http://www.turnkeylinux.org/couchdb" + }, + { + "release": "0", + "app_name": "deki", + "gh_page": "https://github.com/turnkeylinux-apps/deki", + "app_page": "http://www.turnkeylinux.org/deki" + }, + { + "release": "14.1", + "app_name": "django", + "gh_page": "https://github.com/turnkeylinux-apps/django", + "app_page": "http://www.turnkeylinux.org/django" + }, + { + "release": "14.1", + "app_name": "dokuwiki", + "gh_page": "https://github.com/turnkeylinux-apps/dokuwiki", + "app_page": "http://www.turnkeylinux.org/dokuwiki" + }, + { + "release": "14.1", + "app_name": "domain-controller", + "gh_page": "https://github.com/turnkeylinux-apps/domain-controller", + "app_page": "http://www.turnkeylinux.org/domain-controller" + }, + { + "release": "0", + "app_name": "drupal6", + "gh_page": "https://github.com/turnkeylinux-apps/drupal6", + "app_page": "http://www.turnkeylinux.org/drupal6" + }, + { + "release": "14.1", + "app_name": "drupal7", + "gh_page": "https://github.com/turnkeylinux-apps/drupal7", + "app_page": "http://www.turnkeylinux.org/drupal7" + }, + { + "release": "14.1", + "app_name": "drupal8", + "gh_page": "https://github.com/turnkeylinux-apps/drupal8", + "app_page": "" + }, + { + "release": "14.1", + "app_name": "e107", + "gh_page": "https://github.com/turnkeylinux-apps/e107", + "app_page": "http://www.turnkeylinux.org/e107" + }, + { + "release": "13.0", + "app_name": "ejabberd", + "gh_page": "https://github.com/turnkeylinux-apps/ejabberd", + "app_page": "http://www.turnkeylinux.org/ejabberd" + }, + { + "release": "14.1", + "app_name": "elgg", + "gh_page": "https://github.com/turnkeylinux-apps/elgg", + "app_page": "http://www.turnkeylinux.org/elgg" + }, + { + "release": "14.1", + "app_name": "espocrm", + "gh_page": "https://github.com/turnkeylinux-apps/espocrm", + "app_page": "http://www.turnkeylinux.org/espocrm" + }, + { + "release": "14.1", + "app_name": "etherpad", + "gh_page": "https://github.com/turnkeylinux-apps/etherpad", + "app_page": "http://www.turnkeylinux.org/etherpad" + }, + { + "release": "14.1", + "app_name": "ezpublish", + "gh_page": "https://github.com/turnkeylinux-apps/ezpublish", + "app_page": "http://www.turnkeylinux.org/ezpublish" + }, + { + "release": "14.1", + "app_name": "fileserver", + "gh_page": "https://github.com/turnkeylinux-apps/fileserver", + "app_page": "http://www.turnkeylinux.org/fileserver" + }, + { + "release": "14.1", + "app_name": "foodsoft", + "gh_page": "https://github.com/turnkeylinux-apps/foodsoft", + "app_page": "http://www.turnkeylinux.org/foodsoft" + }, + { + "release": "14.1", + "app_name": "gallery", + "gh_page": "https://github.com/turnkeylinux-apps/gallery", + "app_page": "http://www.turnkeylinux.org/gallery" + }, + { + "release": "0", + "app_name": "ghost", + "gh_page": "https://github.com/turnkeylinux-apps/ghost", + "app_page": "https://www.turnkeylinux.org/ghost" + }, + { + "release": "14.1", + "app_name": "gitlab", + "gh_page": "https://github.com/turnkeylinux-apps/gitlab", + "app_page": "http://www.turnkeylinux.org/gitlab" + }, + { + "release": "14.1", + "app_name": "gnusocial", + "gh_page": "https://github.com/turnkeylinux-apps/gnusocial", + "app_page": "http://www.turnkeylinux.org/gnusocial" + }, + { + "release": "14.1", + "app_name": "icescrum", + "gh_page": "https://github.com/turnkeylinux-apps/icescrum", + "app_page": "http://www.turnkeylinux.org/icescrum" + }, + { + "release": "14.1", + "app_name": "jenkins", + "gh_page": "https://github.com/turnkeylinux-apps/jenkins", + "app_page": "http://www.turnkeylinux.org/jenkins" + }, + { + "release": "13.0", + "app_name": "joomla15", + "gh_page": "https://github.com/turnkeylinux-apps/joomla15", + "app_page": "http://www.turnkeylinux.org/joomla15" + }, + { + "release": "13.0", + "app_name": "joomla25", + "gh_page": "https://github.com/turnkeylinux-apps/joomla25", + "app_page": "http://www.turnkeylinux.org/joomla25" + }, + { + "release": "14.1", + "app_name": "joomla3", + "gh_page": "https://github.com/turnkeylinux-apps/joomla3", + "app_page": "http://www.turnkeylinux.org/joomla3" + }, + { + "release": "14.1", + "app_name": "lamp", + "gh_page": "https://github.com/turnkeylinux-apps/lamp", + "app_page": "http://www.turnkeylinux.org/lamp" + }, + { + "release": "14.1", + "app_name": "lapp", + "gh_page": "https://github.com/turnkeylinux-apps/lapp", + "app_page": "http://www.turnkeylinux.org/lapp" + }, + { + "release": "14.1", + "app_name": "laravel", + "gh_page": "https://github.com/turnkeylinux-apps/laravel", + "app_page": "http://www.turnkeylinux.org/laravel" + }, + { + "release": "14.1", + "app_name": "lighttpd-php-fastcgi", + "gh_page": "https://github.com/turnkeylinux-apps/lighttpd-php-fastcgi", + "app_page": "http://www.turnkeylinux.org/lighttpd-php-fastcgi" + }, + { + "release": "14.1", + "app_name": "limesurvey", + "gh_page": "https://github.com/turnkeylinux-apps/limesurvey", + "app_page": "http://www.turnkeylinux.org/limesurvey" + }, + { + "release": "14.1", + "app_name": "lxc", + "gh_page": "https://github.com/turnkeylinux-apps/lxc", + "app_page": "http://www.turnkeylinux.org/lxc" + }, + { + "release": "14.1", + "app_name": "magento", + "gh_page": "https://github.com/turnkeylinux-apps/magento", + "app_page": "http://www.turnkeylinux.org/magento" + }, + { + "release": "14.1", + "app_name": "mahara", + "gh_page": "https://github.com/turnkeylinux-apps/mahara", + "app_page": "http://www.turnkeylinux.org/mahara" + }, + { + "release": "14.1", + "app_name": "mambo", + "gh_page": "https://github.com/turnkeylinux-apps/mambo", + "app_page": "http://www.turnkeylinux.org/mambo" + }, + { + "release": "14.1", + "app_name": "mantis", + "gh_page": "https://github.com/turnkeylinux-apps/mantis", + "app_page": "http://www.turnkeylinux.org/mantis" + }, + { + "release": "0", + "app_name": "mattermost", + "gh_page": "https://github.com/turnkeylinux-apps/mattermost", + "app_page": "https://www.turnkeylinux.org/mattermost" + }, + { + "release": "14.1", + "app_name": "mediaserver", + "gh_page": "https://github.com/turnkeylinux-apps/mediaserver", + "app_page": "" + }, + { + "release": "14.1", + "app_name": "mediawiki", + "gh_page": "https://github.com/turnkeylinux-apps/mediawiki", + "app_page": "http://www.turnkeylinux.org/mediawiki" + }, + { + "release": "14.1", + "app_name": "mibew", + "gh_page": "https://github.com/turnkeylinux-apps/mibew", + "app_page": "http://www.turnkeylinux.org/mibew" + }, + { + "release": "14.1", + "app_name": "moinmoin", + "gh_page": "https://github.com/turnkeylinux-apps/moinmoin", + "app_page": "http://www.turnkeylinux.org/moinmoin" + }, + { + "release": "14.1", + "app_name": "mongodb", + "gh_page": "https://github.com/turnkeylinux-apps/mongodb", + "app_page": "http://www.turnkeylinux.org/mongodb" + }, + { + "release": "14.1", + "app_name": "moodle", + "gh_page": "https://github.com/turnkeylinux-apps/moodle", + "app_page": "http://www.turnkeylinux.org/moodle" + }, + { + "release": "13.0", + "app_name": "movabletype", + "gh_page": "https://github.com/turnkeylinux-apps/movabletype", + "app_page": "http://www.turnkeylinux.org/movabletype" + }, + { + "release": "14.1", + "app_name": "mysql", + "gh_page": "https://github.com/turnkeylinux-apps/mysql", + "app_page": "http://www.turnkeylinux.org/mysql" + }, + { + "release": "14.1", + "app_name": "nginx-php-fastcgi", + "gh_page": "https://github.com/turnkeylinux-apps/nginx-php-fastcgi", + "app_page": "http://www.turnkeylinux.org/nginx-php-fastcgi" + }, + { + "release": "14.1", + "app_name": "nodejs", + "gh_page": "https://github.com/turnkeylinux-apps/nodejs", + "app_page": "http://www.turnkeylinux.org/nodejs" + }, + { + "release": "14.1", + "app_name": "observium", + "gh_page": "https://github.com/turnkeylinux-apps/observium", + "app_page": "http://www.turnkeylinux.org/observium" + }, + { + "release": "14.1", + "app_name": "odoo", + "gh_page": "https://github.com/turnkeylinux-apps/odoo", + "app_page": "" + }, + { + "release": "14.1", + "app_name": "omeka", + "gh_page": "https://github.com/turnkeylinux-apps/omeka", + "app_page": "http://www.turnkeylinux.org/omeka" + }, + { + "release": "14.1", + "app_name": "openldap", + "gh_page": "https://github.com/turnkeylinux-apps/openldap", + "app_page": "http://www.turnkeylinux.org/openldap" + }, + { + "release": "13.0", + "app_name": "openphoto", + "gh_page": "https://github.com/turnkeylinux-apps/openphoto", + "app_page": "http://www.turnkeylinux.org/openphoto" + }, + { + "release": "14.1", + "app_name": "openvpn", + "gh_page": "https://github.com/turnkeylinux-apps/openvpn", + "app_page": "http://www.turnkeylinux.org/openvpn" + }, + { + "release": "14.1", + "app_name": "orangehrm", + "gh_page": "https://github.com/turnkeylinux-apps/orangehrm", + "app_page": "http://www.turnkeylinux.org/orangehrm" + }, + { + "release": "14.1", + "app_name": "oscommerce", + "gh_page": "https://github.com/turnkeylinux-apps/oscommerce", + "app_page": "http://www.turnkeylinux.org/oscommerce" + }, + { + "release": "0", + "app_name": "osqa", + "gh_page": "https://github.com/turnkeylinux-apps/osqa", + "app_page": "http://www.turnkeylinux.org/osqa" + }, + { + "release": "14.1", + "app_name": "otrs", + "gh_page": "https://github.com/turnkeylinux-apps/otrs", + "app_page": "http://www.turnkeylinux.org/otrs" + }, + { + "release": "14.1", + "app_name": "owncloud", + "gh_page": "https://github.com/turnkeylinux-apps/owncloud", + "app_page": "http://www.turnkeylinux.org/owncloud" + }, + { + "release": "14.1", + "app_name": "phpbb", + "gh_page": "https://github.com/turnkeylinux-apps/phpbb", + "app_page": "http://www.turnkeylinux.org/phpbb" + }, + { + "release": "14.1", + "app_name": "phplist", + "gh_page": "https://github.com/turnkeylinux-apps/phplist", + "app_page": "http://www.turnkeylinux.org/phplist" + }, + { + "release": "13.0", + "app_name": "phpnuke", + "gh_page": "https://github.com/turnkeylinux-apps/phpnuke", + "app_page": "http://www.turnkeylinux.org/phpnuke" + }, + { + "release": "14.1", + "app_name": "phreebooks", + "gh_page": "https://github.com/turnkeylinux-apps/phreebooks", + "app_page": "http://www.turnkeylinux.org/phreebooks" + }, + { + "release": "14.1", + "app_name": "piwik", + "gh_page": "https://github.com/turnkeylinux-apps/piwik", + "app_page": "http://www.turnkeylinux.org/piwik" + }, + { + "release": "14.1", + "app_name": "pligg", + "gh_page": "https://github.com/turnkeylinux-apps/pligg", + "app_page": "http://www.turnkeylinux.org/pligg" + }, + { + "release": "14.1", + "app_name": "plone", + "gh_page": "https://github.com/turnkeylinux-apps/plone", + "app_page": "http://www.turnkeylinux.org/plone" + }, + { + "release": "14.1", + "app_name": "postgresql", + "gh_page": "https://github.com/turnkeylinux-apps/postgresql", + "app_page": "http://www.turnkeylinux.org/postgresql" + }, + { + "release": "14.1", + "app_name": "prestashop", + "gh_page": "https://github.com/turnkeylinux-apps/prestashop", + "app_page": "http://www.turnkeylinux.org/prestashop" + }, + { + "release": "14.1", + "app_name": "processmaker", + "gh_page": "https://github.com/turnkeylinux-apps/processmaker", + "app_page": "http://www.turnkeylinux.org/processmaker" + }, + { + "release": "14.1", + "app_name": "projectpier", + "gh_page": "https://github.com/turnkeylinux-apps/projectpier", + "app_page": "http://www.turnkeylinux.org/projectpier" + }, + { + "release": "14.1", + "app_name": "punbb", + "gh_page": "https://github.com/turnkeylinux-apps/punbb", + "app_page": "http://www.turnkeylinux.org/punbb" + }, + { + "release": "14.1", + "app_name": "rails", + "gh_page": "https://github.com/turnkeylinux-apps/rails", + "app_page": "http://www.turnkeylinux.org/rails" + }, + { + "release": "14.1", + "app_name": "redmine", + "gh_page": "https://github.com/turnkeylinux-apps/redmine", + "app_page": "http://www.turnkeylinux.org/redmine" + }, + { + "release": "14.1", + "app_name": "revision-control", + "gh_page": "https://github.com/turnkeylinux-apps/revision-control", + "app_page": "http://www.turnkeylinux.org/revision-control" + }, + { + "release": "14.1", + "app_name": "roundup", + "gh_page": "https://github.com/turnkeylinux-apps/roundup", + "app_page": "http://www.turnkeylinux.org/roundup" + }, + { + "release": "14.1", + "app_name": "sahana-eden", + "gh_page": "https://github.com/turnkeylinux-apps/sahana-eden", + "app_page": "http://www.turnkeylinux.org/sahana-eden" + }, + { + "release": "0", + "app_name": "sencha", + "gh_page": "https://github.com/turnkeylinux-apps/sencha", + "app_page": "http://www.turnkeylinux.org/sencha" + }, + { + "release": "14.1", + "app_name": "silverstripe", + "gh_page": "https://github.com/turnkeylinux-apps/silverstripe", + "app_page": "http://www.turnkeylinux.org/silverstripe" + }, + { + "release": "14.1", + "app_name": "simpleinvoices", + "gh_page": "https://github.com/turnkeylinux-apps/simpleinvoices", + "app_page": "http://www.turnkeylinux.org/simpleinvoices" + }, + { + "release": "14.1", + "app_name": "simplemachines", + "gh_page": "https://github.com/turnkeylinux-apps/simplemachines", + "app_page": "http://www.turnkeylinux.org/simplemachines" + }, + { + "release": "14.1", + "app_name": "sitracker", + "gh_page": "https://github.com/turnkeylinux-apps/sitracker", + "app_page": "http://www.turnkeylinux.org/sitracker" + }, + { + "release": "14.1", + "app_name": "sugarcrm", + "gh_page": "https://github.com/turnkeylinux-apps/sugarcrm", + "app_page": "http://www.turnkeylinux.org/sugarcrm" + }, + { + "release": "14.1", + "app_name": "suitecrm", + "gh_page": "https://github.com/turnkeylinux-apps/suitecrm", + "app_page": "http://www.turnkeylinux.org/suitecrm" + }, + { + "release": "14.1", + "app_name": "symfony", + "gh_page": "https://github.com/turnkeylinux-apps/symfony", + "app_page": "http://www.turnkeylinux.org/symfony" + }, + { + "release": "13.0", + "app_name": "tendenci", + "gh_page": "https://github.com/turnkeylinux-apps/tendenci", + "app_page": "http://www.turnkeylinux.org/tendenci" + }, + { + "release": "14.1", + "app_name": "tkldev", + "gh_page": "https://github.com/turnkeylinux-apps/tkldev", + "app_page": "http://www.turnkeylinux.org/tkldev" + }, + { + "release": "14.1", + "app_name": "tomatocart", + "gh_page": "https://github.com/turnkeylinux-apps/tomatocart", + "app_page": "http://www.turnkeylinux.org/tomatocart" + }, + { + "release": "14.1", + "app_name": "tomcat", + "gh_page": "https://github.com/turnkeylinux-apps/tomcat", + "app_page": "http://www.turnkeylinux.org/tomcat" + }, + { + "release": "14.1", + "app_name": "tomcat-apache", + "gh_page": "https://github.com/turnkeylinux-apps/tomcat-apache", + "app_page": "http://www.turnkeylinux.org/tomcat-apache" + }, + { + "release": "14.1", + "app_name": "torrentserver", + "gh_page": "https://github.com/turnkeylinux-apps/torrentserver", + "app_page": "http://www.turnkeylinux.org/torrentserver" + }, + { + "release": "14.1", + "app_name": "trac", + "gh_page": "https://github.com/turnkeylinux-apps/trac", + "app_page": "http://www.turnkeylinux.org/trac" + }, + { + "release": "14.1", + "app_name": "tracks", + "gh_page": "https://github.com/turnkeylinux-apps/tracks", + "app_page": "http://www.turnkeylinux.org/tracks" + }, + { + "release": "14.1", + "app_name": "twiki", + "gh_page": "https://github.com/turnkeylinux-apps/twiki", + "app_page": "http://www.turnkeylinux.org/twiki" + }, + { + "release": "14.1", + "app_name": "typo3", + "gh_page": "https://github.com/turnkeylinux-apps/typo3", + "app_page": "http://www.turnkeylinux.org/typo3" + }, + { + "release": "14.1", + "app_name": "ushahidi", + "gh_page": "https://github.com/turnkeylinux-apps/ushahidi", + "app_page": "http://www.turnkeylinux.org/ushahidi" + }, + { + "release": "14.1", + "app_name": "vanilla", + "gh_page": "https://github.com/turnkeylinux-apps/vanilla", + "app_page": "http://www.turnkeylinux.org/vanilla" + }, + { + "release": "14.1", + "app_name": "vtiger", + "gh_page": "https://github.com/turnkeylinux-apps/vtiger", + "app_page": "http://www.turnkeylinux.org/vtiger" + }, + { + "release": "14.1", + "app_name": "web2py", + "gh_page": "https://github.com/turnkeylinux-apps/web2py", + "app_page": "http://www.turnkeylinux.org/web2py" + }, + { + "release": "14.1", + "app_name": "wordpress", + "gh_page": "https://github.com/turnkeylinux-apps/wordpress", + "app_page": "http://www.turnkeylinux.org/wordpress" + }, + { + "release": "14.1", + "app_name": "xoops", + "gh_page": "https://github.com/turnkeylinux-apps/xoops", + "app_page": "http://www.turnkeylinux.org/xoops" + }, + { + "release": "14.1", + "app_name": "yiiframework", + "gh_page": "https://github.com/turnkeylinux-apps/yiiframework", + "app_page": "http://www.turnkeylinux.org/yiiframework" + }, + { + "release": "14.1", + "app_name": "zencart", + "gh_page": "https://github.com/turnkeylinux-apps/zencart", + "app_page": "http://www.turnkeylinux.org/zencart" + }, + { + "release": "14.1", + "app_name": "zurmo", + "gh_page": "https://github.com/turnkeylinux-apps/zurmo", + "app_page": "http://www.turnkeylinux.org/zurmo" + } +] \ No newline at end of file diff --git a/contrib/config.json.example b/contrib/config.json.example new file mode 100644 index 0000000..2e65a30 --- /dev/null +++ b/contrib/config.json.example @@ -0,0 +1,3 @@ +{ + "gh_token": "GITHUB_API_TOKEN" +} diff --git a/contrib/get-apps.py b/contrib/get-apps.py new file mode 100755 index 0000000..eba5d27 --- /dev/null +++ b/contrib/get-apps.py @@ -0,0 +1,46 @@ +#!/usr/bin/python + +import json +from octohub.connection import Connection +from octohub.connection import Pager + +app_data = [] +with open("config.json","rb") as fob: + config=json.load(fob) +conn = Connection(config["gh_token"]) +uri = '/users/turnkeylinux-apps/repos' +pager = Pager(conn, uri, {}, max_pages=0) +for response in pager: + data = response.json() + for app in data: + uri = '/repos/turnkeylinux-apps/' + app['name'] + '/tags' + print uri + pager2 = Pager(conn, uri, {}, max_pages=0) + for response2 in pager2: + data2 = response2.json() + release = [0,0] + for entry in data2: + version = entry["name"].split("+") + if len(version) < 2: + version.append(0) + + version[0] = float(version[0]) + version[1] = int(version[1]) + if version > release: + release = version + + if release[1] == 0: + latest_release = str(release[0]) + else: + latest_release = str(release[0]) + '+' + str(release[1]) + + app_data.append({ + "app_name": app['name'], + "release": latest_release, + "app_page": app["homepage"], + "gh_page": app["html_url"] + }) + +with open("apps.json","wb") as fob: + json.dump(app_data, fob, indent=4) + From 0c5952388e16d81a59aa5ae29b511a77b6d528d0 Mon Sep 17 00:00:00 2001 From: Jeremy Davis Date: Fri, 13 Jan 2017 17:43:21 +1100 Subject: [PATCH 4/5] updated .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 5f0a32e..c1f24dc 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ wiki issues contrib/config.json +contrib/*.bak* From 829290ddc8953e4850da75914df1a9ec1de84868 Mon Sep 17 00:00:00 2001 From: Jeremy Davis Date: Fri, 13 Jan 2017 17:44:15 +1100 Subject: [PATCH 5/5] updated get-apps.py - now checks website version, proxmox version and mirror files --- contrib/get-apps.py | 179 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 169 insertions(+), 10 deletions(-) diff --git a/contrib/get-apps.py b/contrib/get-apps.py index eba5d27..fcb9267 100755 --- a/contrib/get-apps.py +++ b/contrib/get-apps.py @@ -1,20 +1,111 @@ #!/usr/bin/python import json +import requests +import os +import datetime +from lxml import html from octohub.connection import Connection from octohub.connection import Pager +def getTimestamp(): + timestamp = datetime.datetime.now().strftime('%Y-%m-%d_%H-%M-%S') + return timestamp + +def checkAndBackupFile(file): + if os.path.isfile(file): + backup_name = file+".bak."+getTimestamp() + print(file+" already exists. File renamed to "+backup_name) + os.rename(file, backup_name) + +def checkWebStatus(uri): + try: + r = requests.head(uri, allow_redirects=True) + return_code = { + 200: "stable", + 403: "deprecated", + 404: "not_found" + } + code = return_code.get(r.status_code, "error: unknown code "+str(r.status_code)) + return code + except requests.ConnectionError: + print("failed to connect to "+uri) + +def getURIText(uri, type): + try: + r = requests.get(uri) + if type == "text": + file = r.text + elif type == "web": + file = html.fromstring(r.content) + else: + raise ValueError("type not known", type) + return file + except requests.ConnectionError: + print("failed to connect to "+uri) + +def checkChangelog(app): + uri = "https://raw.githubusercontent.com/turnkeylinux-apps/"+app+"/master/changelog" + changelog = getURIText(uri, "text") + head = changelog.split('\n', 1)[0] + changelog_version = head.split(app+'-')[1].split()[0] + changelog_revision = head.split(app+'-')[1].split()[1] + return changelog_version, changelog_revision + +def checkAppPage(app): + uri = "https://www.turnkeylinux.org/"+app + page = getURIText(uri, "web") + web_version = page.xpath('//*/div[2]/div[3]/div[2]/span/text()') + return web_version[0] + +def checkMirror(app, version, arch, build): + uri = "http://mirror.turnkeylinux.org/turnkeylinux/images/"+build + if build == "proxmox": + #assumes only amd64 proxmox builds + uri += "/debian-8-turnkey-"+app+"_"+version+"-1_amd64.tar.gz" + elif build in ("iso", "ova"): + uri += "/turnkey-"+app+"-"+version+"-jessie-"+arch+"."+build + else: + raise ValueError("build not known", build) + + code = checkWebStatus(uri) + + if code == "stable": + return "yes" + else: + return "not found" + +def checkPVEIndex(app, pve_app_index): + pve_index_version = pve_download = None + for item in pve_app_index.split("\n\n"): + if app in item: + for line in item.split("\n"): + if "Version" in line: + pve_index_version = line.split(": ")[1].split("-")[0] + if "Location" in line: + pve_download = line.split(": ")[1] + return pve_index_version, pve_download + app_data = [] +appliance_list = "" +pve_app_index = getURIText('http://mirror.turnkeylinux.org/turnkeylinux/metadata/pve/aplinfo.dat', 'text') + +checkAndBackupFile('appliance.list') +checkAndBackupFile('apps.json') + with open("config.json","rb") as fob: config=json.load(fob) + conn = Connection(config["gh_token"]) uri = '/users/turnkeylinux-apps/repos' pager = Pager(conn, uri, {}, max_pages=0) for response in pager: + notes = "" data = response.json() for app in data: + notes = "" + print("Processing appliance: "+app['name']) uri = '/repos/turnkeylinux-apps/' + app['name'] + '/tags' - print uri pager2 = Pager(conn, uri, {}, max_pages=0) for response2 in pager2: data2 = response2.json() @@ -30,17 +121,85 @@ release = version if release[1] == 0: - latest_release = str(release[0]) + latest_tag = str(release[0]) else: - latest_release = str(release[0]) + '+' + str(release[1]) + latest_tag = str(release[0]) + '+' + str(release[1]) + + if not app['homepage']: + status = "error" + notes += "|no homepage listed on GH repo|" + app['homepage'] = "https://www/turnkeylinux.org/"+app['name'] - app_data.append({ - "app_name": app['name'], - "release": latest_release, - "app_page": app["homepage"], - "gh_page": app["html_url"] - }) + status = checkWebStatus(app['homepage']) + + downloads = {} + + if status == "stable" and app['name'] != "bitkey": + + appliance_list += app['name'] + "\n" + + changelog_ver, changelog_rev = checkChangelog(app['name']) + + if changelog_ver != latest_tag: + status = "error" + notes += "|changelog version and tag version don't match|" + + for build in ("iso", "proxmox", "ova"): + if build == "iso": + for arch in ("amd64", "i386"): + downloads[build+"-"+arch] = checkMirror(app['name'], latest_tag, arch, build) + else: + downloads[build] = checkMirror(app['name'], latest_tag, "amd64", build) + if downloads["iso-amd64"] == "not found": + status = "error" + notes += "|iso-amd64 not found on mirror|" + + pve_index_version, pve_download = checkPVEIndex(app['name'], pve_app_index) + + if pve_index_version: + if pve_index_version != latest_tag: + status = "error" + notes += "|pve index version does not match latest tag|" + if pve_index_version != changelog_ver: + status = "error" + notes += "|pve index version does not match latest changelog entry|" + if checkWebStatus(pve_download) != "stable": + status = "error" + notes += "|appliance from pve index not found on mirror|" + else: + pve_index_version = "not listed" + notes += "|appliance not listed in pve index|" + + app_page_version = checkAppPage(app['name']) + + else: + notes = "|skipped sections as noted|" + for key in ['iso-amd64', 'iso-i386', 'proxmox', 'ova']: + downloads[key] = "skipped" + pve_index_version = "skipped" + app_page_version = "skipped" + + app_data.append({ + "app name": app['name'], + "app github tag": latest_tag, + "app page version": app_page_version, + "app status": status, + "app webpage": app["homepage"], + "changelog version": changelog_ver, + "changelog revison": changelog_rev, + "file: iso-amd64": downloads["iso-amd64"], + "file: iso-i386": downloads["iso-i386"], + "file: proxmox": downloads["proxmox"], + "proxmox index version": pve_index_version, + "file: ova-amd64": downloads["ova"], + "github page": app["html_url"], + "notes": notes, + "timestamp": getTimestamp() + }) with open("apps.json","wb") as fob: - json.dump(app_data, fob, indent=4) + json.dump(app_data, fob, indent=4, sort_keys=True) + +with open("appliance.list", "wb") as fob: + fob.write(appliance_list)