Merge pull request #74 from ioet/feature/add-migrations-support#36

Closes #36 Allow to run migrations

Author: EliuX

.env.template

@@ -4,11 +4,11 @@ export FLASK_APP=time_tracker_api
 
 # Common attributes
 ## In case you use an Azure SQL database, you must specify the database connection URI. Check out the README.md for more details
-#export DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
+#export SQL_DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
 
 ## For Azure Cosmos DB
 export DATABASE_ACCOUNT_URI=https://<project_db_name>.documents.azure.com:443
 export DATABASE_MASTER_KEY=<db_master_key>
 export DATABASE_NAME=<db_name>
 ### or
-# export DATABASE_URI=AccountEndpoint=<ACCOUNT_URI>;AccountKey=<ACCOUNT_KEY>
+# export COSMOS_DATABASE_URI=AccountEndpoint=<ACCOUNT_URI>;AccountKey=<ACCOUNT_KEY>

.gitignore

@@ -28,5 +28,8 @@ htmlcov/
 timetracker-api-postman-collection.json
 swagger.json
 
-# Ignore any SQLite generated database
+# SQLite databases
 *.db
+
+# Local migration files
+migration_status.csv

README.md

@@ -60,10 +60,10 @@ Driver=/usr/local/lib/libmsodbcsql.17.dylib
 UsageCount=2
 ```
 
-Then specify the driver name, in this case _DBC Driver 17 for SQL Server_ in the `DATABASE_URI`, e.g.:
+Then specify the driver name, in this case _DBC Driver 17 for SQL Server_ in the `SQL_DATABASE_URI`, e.g.:
 
 ```.dotenv
-DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
+SQL_DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
 ```
 
 To troubleshoot issues regarding this part please check out:
@@ -115,7 +115,7 @@ tests as [system testing](https://en.wikipedia.org/wiki/System_testing):
 python3 -m pytest -v
 ```
 
-The database tests will be done in the table `tests` of the database specified by the variable `DATABASE_URI`. If this
+The database tests will be done in the table `tests` of the database specified by the variable `SQL_DATABASE_URI`. If this
 variable is not specified it will automatically connect to SQLite database in-memory. This will do, because we are using 
 [SQL Alchemy](https://www.sqlalchemy.org/features.html) to be able connect to any SQL database maintaining the same 
 codebase.
@@ -190,6 +190,36 @@ docker run -p 5000:5000 time_tracker_api:local
 
 3. Visit `127.0.0.1:5000`
 
+## Migrations
+Looking for a DB-agnostic migration tool, the only choice I found was [migrate-anything](https://pypi.org/project/migrate-anything/).
+An specific requirement file was created to run the migrations in `requirements/migrations.txt`. This way we do not mix
+any possible vulnerable dependency brought by these dependencies to the environment `prod`. Therefore the dependencies
+to run the migrations shall be installed this way:
+
+```bash
+pip install -r requirements/<app>/prod.txt
+pip install -r requirements/migrations.txt
+```
+
+All the migrations will be handled and created in the python package `migrations`. In order to create a migration we 
+must do it manually (for now) and prefixed by a number, e.g. `migrations/01-initialize-db.py` in order to warranty the 
+order of execution alphabetically.
+Inside every migration there is an `up` and `down` method. The `down` method is executed from the persisted migration in
+the database. Whe a `down` logic that used external dependencies was tested it failed, whilst I put that same logic in 
+the an `up` method it run correctly. In general the library seems to present [design issues](https://github.com/Lieturd/migrate-anything/issues/3).
+Therefore, it is recommended to apply changes just in one direction: `up`.
+For more information, please check out [some examples](https://github.com/Lieturd/migrate-anything/tree/master/examples) 
+that illustrates the usage of this migration tool.
+
+Basically, for running the migrations you must execute
+
+```bash
+migrate-anything migrations
+```
+
+They will be automatically run during the Continuous Deployment process.
+
+
 ## Built with
 - [Python version 3](https://www.python.org/download/releases/3.0/) as backend programming language. Strong typing for 
 the win.

commons/data_access_layer/cosmos_db.py

@@ -1,20 +1,28 @@
 import dataclasses
+import logging
 import uuid
 from typing import Callable
 
 import azure.cosmos.cosmos_client as cosmos_client
 import azure.cosmos.exceptions as exceptions
-from azure.cosmos import ContainerProxy
+from azure.cosmos import ContainerProxy, PartitionKey
 from flask import Flask
 
 
 class CosmosDBFacade:
-    def __init__(self, app: Flask):  # pragma: no cover
-        self.app = app
+    def __init__(self, client, db_id: str, logger=None):  # pragma: no cover
+        self.client = client
+        self.db = self.client.get_database_client(db_id)
+        if logger is None:
+            self.logger = logging.getLogger(CosmosDBFacade.__name__)
+        else:
+            self.logger = logger
 
-        db_uri = app.config.get('DATABASE_URI')
+    @classmethod
+    def from_flask_config(cls, app: Flask):
+        db_uri = app.config.get('COSMOS_DATABASE_URI')
         if db_uri is None:
-            app.logger.warn("DATABASE_URI was not found. Looking for alternative variables.")
+            app.logger.warn("COSMOS_DATABASE_URI was not found. Looking for alternative variables.")
             account_uri = app.config.get('DATABASE_ACCOUNT_URI')
             if account_uri is None:
                 raise EnvironmentError("DATABASE_ACCOUNT_URI is not defined in the environment")
@@ -23,31 +31,26 @@ def __init__(self, app: Flask):  # pragma: no cover
             if master_key is None:
                 raise EnvironmentError("DATABASE_MASTER_KEY is not defined in the environment")
 
-            self.client = cosmos_client.CosmosClient(account_uri, {'masterKey': master_key},
-                                                     user_agent="CosmosDBDotnetQuickstart",
-                                                     user_agent_overwrite=True)
+            client = cosmos_client.CosmosClient(account_uri, {'masterKey': master_key},
+                                                user_agent="CosmosDBDotnetQuickstart",
+                                                user_agent_overwrite=True)
         else:
-            self.client = cosmos_client.CosmosClient.from_connection_string(db_uri)
+            client = cosmos_client.CosmosClient.from_connection_string(db_uri)
 
         db_id = app.config.get('DATABASE_NAME')
         if db_id is None:
             raise EnvironmentError("DATABASE_NAME is not defined in the environment")
 
-        self.db = self.client.get_database_client(db_id)
+        return cls(client, db_id, logger=app.logger)
 
     def create_container(self, container_definition: dict):
-        try:
-            return self.db.create_container(**container_definition)
+        return self.db.create_container(**container_definition)
 
-        except exceptions.CosmosResourceExistsError:  # pragma: no cover
-            self.app.logger.info('Container with id \'{0}\' was found'.format(container_definition["id"]))
+    def create_container_if_not_exists(self, container_definition: dict):
+        return self.db.create_container_if_not_exists(**container_definition)
 
     def delete_container(self, container_id: str):
-        try:
-            return self.db.delete_container(container_id)
-
-        except exceptions.CosmosHttpResponseError:  # pragma: no cover
-            self.app.logger.info('Container with id \'{0}\' was not deleted'.format(container_id))
+        return self.db.delete_container(container_id)
 
 
 cosmos_helper: CosmosDBFacade = None
@@ -61,8 +64,13 @@ def __init__(self, data):
                 setattr(self, k, v)
 
 
+def partition_key_attribute(pk: PartitionKey) -> str:
+    return pk.path.strip('/')
+
+
 class CosmosDBRepository:
     def __init__(self, container_id: str,
+                 partition_key_attribute: str,
                  mapper: Callable = None,
                  custom_cosmos_helper: CosmosDBFacade = None):
         global cosmos_helper
@@ -71,12 +79,16 @@ def __init__(self, container_id: str,
             raise ValueError("The cosmos_db module has not been initialized!")
         self.mapper = mapper
         self.container: ContainerProxy = self.cosmos_helper.db.get_container_client(container_id)
+        self.partition_key_attribute: str = partition_key_attribute
 
     @classmethod
     def from_definition(cls, container_definition: dict,
                         mapper: Callable = None,
                         custom_cosmos_helper: CosmosDBFacade = None):
-        return cls(container_definition['id'], mapper, custom_cosmos_helper)
+        pk_attrib = partition_key_attribute(container_definition['partition_key'])
+        return cls(container_definition['id'], pk_attrib,
+                   mapper=mapper,
+                   custom_cosmos_helper=custom_cosmos_helper)
 
     def create(self, data: dict, mapper: Callable = None):
         function_mapper = self.get_mapper_or_dict(mapper)
@@ -93,11 +105,12 @@ def find_all(self, partition_key_value: str, max_count=None, offset=0,
         max_count = self.get_page_size_or(max_count)
         result = self.container.query_items(
             query="""
-            SELECT * FROM c WHERE c.tenant_id=@tenant_id AND {visibility_condition}
+            SELECT * FROM c WHERE c.{partition_key_attribute}=@partition_key_value AND {visibility_condition}
             OFFSET @offset LIMIT @max_count
-            """.format(visibility_condition=self.create_sql_condition_for_visibility(visible_only)),
+            """.format(partition_key_attribute=self.partition_key_attribute,
+                       visibility_condition=self.create_sql_condition_for_visibility(visible_only)),
             parameters=[
-                {"name": "@tenant_id", "value": partition_key_value},
+                {"name": "@partition_key_value", "value": partition_key_value},
                 {"name": "@offset", "value": offset},
                 {"name": "@max_count", "value": max_count},
             ],
@@ -122,6 +135,9 @@ def delete(self, id: str, partition_key_value: str, mapper: Callable = None):
             'deleted': str(uuid.uuid4())
         }, partition_key_value, visible_only=True, mapper=mapper)
 
+    def delete_permanently(self, id: str, partition_key_value: str) -> None:
+        self.container.delete_item(id, partition_key_value)
+
     def check_visibility(self, item, throw_not_found_if_deleted):
         if throw_not_found_if_deleted and item.get('deleted') is not None:
             raise exceptions.CosmosResourceNotFoundError(message='Deleted item',
@@ -146,4 +162,4 @@ def get_page_size_or(self, custom_page_size: int) -> int:
 
 def init_app(app: Flask) -> None:
     global cosmos_helper
-    cosmos_helper = CosmosDBFacade(app)
+    cosmos_helper = CosmosDBFacade.from_flask_config(app)

migrations/01-initialize-db.py

@@ -0,0 +1,35 @@
+def up():
+    from commons.data_access_layer.cosmos_db import cosmos_helper
+    import azure.cosmos.exceptions as exceptions
+    from . import app
+
+    app.logger.info("Creating TimeTracker initial containers...")
+
+    try:
+        app.logger.info('- Project')
+        from time_tracker_api.projects.projects_model import container_definition as project_definition
+        cosmos_helper.create_container(project_definition)
+
+        app.logger.info('- Project type')
+        from time_tracker_api.project_types.project_types_model import container_definition as project_type_definition
+        cosmos_helper.create_container(project_type_definition)
+
+        app.logger.info('- Activity')
+        from time_tracker_api.activities.activities_model import container_definition as activity_definition
+        cosmos_helper.create_container(activity_definition)
+
+        app.logger.info('- Customer')
+        from time_tracker_api.customers.customers_model import container_definition as customer_definition
+        cosmos_helper.create_container(customer_definition)
+
+        app.logger.info('- Time entry')
+        from time_tracker_api.time_entries.time_entries_model import container_definition as time_entry_definition
+        cosmos_helper.create_container(time_entry_definition)
+    except exceptions.CosmosResourceExistsError as e:
+        app.logger.warning("Unexpected error while creating initial database schema: %s" % e.message)
+
+    app.logger.info("Done!")
+
+
+def down():
+    print("Not implemented!")

migrations/__init__.py

@@ -0,0 +1,77 @@
+from azure.cosmos import PartitionKey
+from migrate_anything import configure
+from migrate_anything.storage import Storage
+
+from time_tracker_api import create_app
+
+
+class CustomStorage(object):
+    def __init__(self, file):
+        self.file = file
+
+    def save_migration(self, name, code):
+        with open(self.file, "a", encoding="utf-8") as file:
+            file.write("{},{}\n".format(name, code))
+
+    def list_migrations(self):
+        try:
+            with open(self.file, encoding="utf-8") as file:
+                return [
+                    line.split(",")
+                    for line in file.readlines()
+                    if line.strip()  # Skip empty lines
+                ]
+        except FileNotFoundError:
+            return []
+
+    def remove_migration(self, name):
+        migrations = [
+            migration for migration in self.list_migrations() if migration[0] != name
+        ]
+
+        with open(self.file, "w", encoding="utf-8") as file:
+            for row in migrations:
+                file.write("{},{}\n".format(*row))
+
+
+app = create_app('time_tracker_api.config.CLIConfig')
+from commons.data_access_layer.cosmos_db import cosmos_helper, init_app, CosmosDBRepository
+
+if cosmos_helper is None:
+    init_app(app)
+    from commons.data_access_layer.cosmos_db import cosmos_helper
+
+
+class CosmosDBStorage(Storage):
+    def __init__(self, collection_id, app_id):
+        self.collection_id = collection_id
+        self.app_id = app_id
+        migrations_definition = {
+            'id': collection_id,
+            'partition_key': PartitionKey(path='/app_id'),
+            'unique_key_policy': {
+                'uniqueKeys': [
+                    {'paths': ['/name']},
+                ]
+            }
+        }
+        cosmos_helper.create_container_if_not_exists(migrations_definition)
+        self.repository = CosmosDBRepository.from_definition(migrations_definition)
+
+    def save_migration(self, name, code):
+        self.repository.create({"id": name,
+                                "name": name,
+                                "code": code,
+                                "app_id": self.app_id})
+
+    def list_migrations(self):
+        migrations = self.repository.find_all(self.app_id)
+        return [
+            [item['name'], item['code']] for item in migrations
+        ]
+
+    def remove_migration(self, name):
+        self.repository.delete_permanently(name, self.app_id)
+
+
+configure(storage=CosmosDBStorage("migrations", "time-tracker-api"))

requirements/migrations.txt

@@ -0,0 +1,6 @@
+# requirements/migrations.txt
+
+# For running any kind of data migration
+
+# Migration tool
+migrate-anything==0.1.6

tests/commons/data_access_layer/cosmos_db_test.py

@@ -507,3 +507,32 @@ def test_partial_update_should_not_find_element_that_is_already_deleted(
     except Exception as e:
         assert type(e) is CosmosResourceNotFoundError
         assert e.status_code == 404
+
+
+def test_delete_permanently_with_invalid_id_should_fail(
+        cosmos_db_repository: CosmosDBRepository,
+        sample_item: dict):
+    try:
+        cosmos_db_repository.delete_permanently(fake.uuid4(), sample_item['tenant_id'])
+        fail('It should have not found the deleted item')
+    except Exception as e:
+        assert type(e) is CosmosResourceNotFoundError
+        assert e.status_code == 404
+
+
+def test_delete_permanently_with_valid_id_should_succeed(
+        cosmos_db_repository: CosmosDBRepository,
+        sample_item: dict):
+    found_item = cosmos_db_repository.find(sample_item['id'], sample_item['tenant_id'])
+
+    assert found_item is not None
+    assert found_item['id'] == sample_item['id']
+
+    cosmos_db_repository.delete_permanently(sample_item['id'], sample_item['tenant_id'])
+
+    try:
+        cosmos_db_repository.find(sample_item['id'], sample_item['tenant_id'])
+        fail('It should have not found the deleted item')
+    except Exception as e:
+        assert type(e) is CosmosResourceNotFoundError
+        assert e.status_code == 404

tests/conftest.py

@@ -80,7 +80,7 @@ def cosmos_db_repository(app: Flask, cosmos_db_model) -> CosmosDBRepository:
             from commons.data_access_layer.cosmos_db import cosmos_helper
 
         app.logger.info("Creating Cosmos DB test models...")
-        cosmos_helper.create_container(cosmos_db_model)
+        cosmos_helper.create_container_if_not_exists(cosmos_db_model)
         app.logger.info("Cosmos DB test models created!")
 
         yield CosmosDBRepository.from_definition(cosmos_db_model)