Create Azure functions project time tracker events #53 #101 #102

.gitignore

@@ -33,3 +33,6 @@ swagger.json
 
 # Local migration files
 migration_status.csv
+
+# Mac
+.DS_Store

README.md

@@ -16,6 +16,7 @@ Be sure you have installed in your system
 - [Python version 3](https://www.python.org/download/releases/3.0/) in your path. It will install
 automatically [pip](https://pip.pypa.io/en/stable/) as well.
 - A virtual environment, namely [venv](https://docs.python.org/3/library/venv.html).
+- Optionally for running Azure functions locally: [Azure functions core tool](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=macos%2Ccsharp%2Cbash).
 
 ### Setup
 - Create and activate the environment,
@@ -38,14 +39,17 @@ automatically [pip](https://pip.pypa.io/en/stable/) as well.
     python3 -m pip install -r requirements/<app>/<stage>.txt
     ```
 
-    Where <app> is one of the executable app namespace, e.g. `time_tracker_api`.
+    Where <app> is one of the executable app namespace, e.g. `time_tracker_api` or `time_tracker_events`.
     The `stage` can be 
 
     * `dev`: Used for working locally
     * `prod`: For anything deployed
 
 
-    Remember to do it with Python 3.
+Remember to do it with Python 3.
+
+Bear in mind that the requirements for `time_tracker_events`, must be located on its local requirements.txt, by 
+    [convention](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python#folder-structure).
 
 - Run `pre-commit install`. For more details, check out Development > Git hooks.
 
@@ -66,6 +70,40 @@ automatically [pip](https://pip.pypa.io/en/stable/) as well.
 - Open `http://127.0.0.1:5000/` in a browser. You will find in the presented UI 
 a link to the swagger.json with the definition of the api.
 
+#### Handling Cosmos DB triggers for creating events with time_tracker_events
+The project `time_tracker_events` is an Azure Function project. Its main responsibility is to respond to calls related to
+events, like those [triggered by Change Feed](https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed-functions).
+Every time a write action (`create`, `update`, `soft-delete`) is done by CosmosDB, thanks to [bindings](https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb?toc=%2Fazure%2Fcosmos-db%2Ftoc.json&bc=%2Fazure%2Fcosmos-db%2Fbreadcrumb%2Ftoc.json&tabs=csharp)
+these functions will be called. You can also run them in your local machine:
+
+- You must have the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli?view=azure-cli-latest)
+and the [Azure Functions Core Tools](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=macos%2Ccsharp%2Cbash)
+installed in your local machine.
+- Be sure to [authenticate](https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli?view=azure-cli-latest)
+with the Azure CLI if you are not.
+```bash
+az login
+```
+-  Execute the project
+```bash
+cd time_tracker_events
+source run.sh
+```
+You will see that a large console log will appear ending with a message like
+```log
+Now listening on: http://0.0.0.0:7071
+Application started. Press Ctrl+C to shut down.
+```
+- Now you are ready to start generating events. Just execute any change in your API and you will see how logs are being
+generated by the console app you ran before. For instance, this is the log generated when I restarted a time entry:
+```log
+[04/30/2020 14:42:12] Executing 'Functions.handle_time_entry_events_trigger' (Reason='New changes on collection time_entry at 2020-04-30T14:42:12.1465310Z', Id=3da87e53-0434-4ff2-8db3-f7c051ccf9fd)
+[04/30/2020 14:42:12]  INFO: Received FunctionInvocationRequest, request ID: 578e5067-b0c0-42b5-a1a4-aac858ea57c0, function ID: c8ac3c4c-fefd-4db9-921e-661b9010a4d9, invocation ID: 3da87e53-0434-4ff2-8db3-f7c051ccf9fd
+[04/30/2020 14:42:12]  INFO: Successfully processed FunctionInvocationRequest, request ID: 578e5067-b0c0-42b5-a1a4-aac858ea57c0, function ID: c8ac3c4c-fefd-4db9-921e-661b9010a4d9, invocation ID: 3da87e53-0434-4ff2-8db3-f7c051ccf9fd
+[04/30/2020 14:42:12] {"id": "9ac108ff-c24d-481e-9c61-b8a3a0737ee8", "project_id": "c2e090fb-ae8b-4f33-a9b8-2052d67d916b", "start_date": "2020-04-28T15:20:36.006Z", "tenant_id": "cc925a5d-9644-4a4f-8d99-0bee49aadd05", "owner_id": "709715c1-6d96-4ecc-a951-b628f2e7d89c", "end_date": null, "_last_event_ctx": {"user_id": "709715c1-6d96-4ecc-a951-b628f2e7d89c", "tenant_id": "cc925a5d-9644-4a4f-8d99-0bee49aadd05", "action": "update", "description": "Restart time entry", "container_id": "time_entry", "session_id": null}, "description": "Changing my description for testing Change Feed", "_metadata": {}}
+[04/30/2020 14:42:12] Executed 'Functions.handle_time_entry_events_trigger' (Succeeded, Id=3da87e53-0434-4ff2-8db3-f7c051ccf9fd)
+```
+
 ### Security
 In this API we are requiring authenticated users using JWT. To do so, we are using the library 
 [PyJWT](https://pypi.org/project/PyJWT/), so in every request to the API we expect a header `Authorization` with a format 
@@ -99,6 +137,16 @@ following notes regarding to the manipulation of the data from and towards the A
 - The [recommended](https://docs.microsoft.com/en-us/azure/cosmos-db/working-with-dates#storing-datetimes) format for 
 DateTime strings in Azure Cosmos DB is `YYYY-MM-DDThh:mm:ss.fffffffZ` which follows the ISO 8601 **UTC standard**.
 
+The Azure function project `time_tracker_events` also have some constraints to have into account. It is recommended that
+you read the [Azure Functions Python developer guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python#folder-structure).
+
+If you require to deploy `time_tracker_events` from your local machine to Azure Functions, you can execute:
+
+```bash
+func azure functionapp publish time-tracker-events  --build local
+```
+
+
 ## Development
 
 ### Git hooks
@@ -254,6 +302,8 @@ the win.
 - [Swagger](https://swagger.io/) for documentation and standardization, taking into account the
 [API import restrictions and known issues](https://docs.microsoft.com/en-us/azure/api-management/api-management-api-import-restrictions)
 in Azure.
+- [Azure Functions bindings](https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb?toc=%2Fazure%2Fcosmos-db%2Ftoc.json&bc=%2Fazure%2Fcosmos-db%2Fbreadcrumb%2Ftoc.json&tabs=csharp)
+for making `time_tracker_events` to handle the triggers [generated by our Cosmos DB database throw Change Feed](https://docs.microsoft.com/bs-latn-ba/azure/cosmos-db/change-feed-functions).
 
 ## License
 

commons/data_access_layer/cosmos_db.py

@@ -10,8 +10,7 @@
 from flask import Flask
 from werkzeug.exceptions import HTTPException
 
-from commons.data_access_layer.database import CRUDDao
-from time_tracker_api.security import current_user_tenant_id
+from commons.data_access_layer.database import CRUDDao, EventContext
 
 
 class CosmosDBFacade:
@@ -140,23 +139,36 @@ def replace_empty_value_per_none(item_data: dict) -> dict:
             if isinstance(v, str) and len(v) == 0:
                 item_data[k] = None
 
-    def create(self, data: dict, mapper: Callable = None):
-        self.on_create(data)
+    @staticmethod
+    def attach_context(data: dict, event_context: EventContext):
+        data["_last_event_ctx"] = {
+            "user_id": event_context.user_id,
+            "tenant_id": event_context.tenant_id,
+            "action": event_context.action,
+            "description": event_context.description,
+            "container_id": event_context.container_id,
+            "session_id": event_context.session_id,
+        }
+
+    def create(self, data: dict, event_context: EventContext, mapper: Callable = None):
+        self.on_create(data, event_context)
         function_mapper = self.get_mapper_or_dict(mapper)
+        self.attach_context(data, event_context)
         return function_mapper(self.container.create_item(body=data))
 
-    def find(self, id: str, partition_key_value, peeker: 'function' = None,
+    def find(self, id: str, event_context: EventContext, peeker: 'function' = None,
              visible_only=True, mapper: Callable = None):
+        partition_key_value = self.find_partition_key_value(event_context)
         found_item = self.container.read_item(id, partition_key_value)
         if peeker:
             peeker(found_item)
 
         function_mapper = self.get_mapper_or_dict(mapper)
         return function_mapper(self.check_visibility(found_item, visible_only))
 
-    def find_all(self, partition_key_value: str, conditions: dict = {}, max_count=None, offset=0,
-                 visible_only=True, mapper: Callable = None):
-        # TODO Use the tenant_id param and change container alias
+    def find_all(self, event_context: EventContext, conditions: dict = {}, max_count=None,
+                 offset=0, visible_only=True, mapper: Callable = None):
+        partition_key_value = self.find_partition_key_value(event_context)
         max_count = self.get_page_size_or(max_count)
         params = [
             {"name": "@partition_key_value", "value": partition_key_value},
@@ -179,27 +191,31 @@ def find_all(self, partition_key_value: str, conditions: dict = {}, max_count=No
         function_mapper = self.get_mapper_or_dict(mapper)
         return list(map(function_mapper, result))
 
-    def partial_update(self, id: str, changes: dict, partition_key_value: str,
+    def partial_update(self, id: str, changes: dict, event_context: EventContext,
                        peeker: 'function' = None, visible_only=True, mapper: Callable = None):
-        item_data = self.find(id, partition_key_value, peeker=peeker,
-                              visible_only=visible_only, mapper=dict)
+        item_data = self.find(id, event_context, peeker=peeker, visible_only=visible_only, mapper=dict)
         item_data.update(changes)
-        return self.update(id, item_data, mapper=mapper)
+        return self.update(id, item_data, event_context, mapper=mapper)
 
-    def update(self, id: str, item_data: dict, mapper: Callable = None):
-        self.on_update(item_data)
+    def update(self, id: str, item_data: dict, event_context: EventContext,
+               mapper: Callable = None):
+        self.on_update(item_data, event_context)
         function_mapper = self.get_mapper_or_dict(mapper)
+        self.attach_context(item_data, event_context)
         return function_mapper(self.container.replace_item(id, body=item_data))
 
-    def delete(self, id: str, partition_key_value: str,
+    def delete(self, id: str, event_context: EventContext,
                peeker: 'function' = None, mapper: Callable = None):
         return self.partial_update(id, {
-            'deleted': str(uuid.uuid4())
-        }, partition_key_value, peeker=peeker, visible_only=True, mapper=mapper)
+            'deleted': generate_uuid4()
+        }, event_context, peeker=peeker, visible_only=True, mapper=mapper)
 
     def delete_permanently(self, id: str, partition_key_value: str) -> None:
         self.container.delete_item(id, partition_key_value)
 
+    def find_partition_key_value(self, event_context: EventContext):
+        return getattr(event_context, self.partition_key_attribute)
+
     def get_mapper_or_dict(self, alternative_mapper: Callable) -> Callable:
         return alternative_mapper or self.mapper or dict
 
@@ -208,13 +224,15 @@ def get_page_size_or(self, custom_page_size: int) -> int:
         # or any other repository for the settings
         return custom_page_size or 100
 
-    def on_create(self, new_item_data: dict):
+    def on_create(self, new_item_data: dict, event_context: EventContext):
         if new_item_data.get('id') is None:
             new_item_data['id'] = generate_uuid4()
 
+        new_item_data[self.partition_key_attribute] = self.find_partition_key_value(event_context)
+
         self.replace_empty_value_per_none(new_item_data)
 
-    def on_update(self, update_item_data: dict):
+    def on_update(self, update_item_data: dict, event_context: EventContext):
         pass
 
     def create_sql_order_clause(self):
@@ -228,27 +246,28 @@ def __init__(self, repository: CosmosDBRepository):
         self.repository = repository
 
     def get_all(self, conditions: dict = {}) -> list:
-        return self.repository.find_all(partition_key_value=self.partition_key_value,
-                                        conditions=conditions)
+        event_ctx = self.create_event_context("read-many")
+        return self.repository.find_all(event_ctx, conditions=conditions)
 
     def get(self, id):
-        return self.repository.find(id, partition_key_value=self.partition_key_value)
+        event_ctx = self.create_event_context("read")
+        return self.repository.find(id, event_ctx)
 
     def create(self, data: dict):
-        data[self.repository.partition_key_attribute] = self.partition_key_value
-        return self.repository.create(data)
+        event_ctx = self.create_event_context("create")
+        return self.repository.create(data, event_ctx)
 
     def update(self, id, data: dict):
-        return self.repository.partial_update(id,
-                                              changes=data,
-                                              partition_key_value=self.partition_key_value)
+        event_ctx = self.create_event_context("update")
+        return self.repository.partial_update(id, data, event_ctx)
 
     def delete(self, id):
-        self.repository.delete(id, partition_key_value=self.partition_key_value)
+        event_ctx = self.create_event_context("delete")
+        self.repository.delete(id, event_ctx)
 
-    @property
-    def partition_key_value(self):
-        return current_user_tenant_id()
+    def create_event_context(self, action: str = None, description: str = None):
+        return EventContext(self.repository.container.id, action,
+                            description=description)
 
 
 class CustomError(HTTPException):

commons/data_access_layer/database.py

@@ -7,8 +7,6 @@
 """
 import abc
 
-from flask import Flask
-
 COMMENTS_MAX_LENGTH = 500
 ID_MAX_LENGTH = 64
 
@@ -35,15 +33,36 @@ def delete(self, id):
         raise NotImplementedError  # pragma: no cover
 
 
-def init_app(app: Flask) -> None:
-    init_cosmos_db(app)
+class EventContext():
+    def __init__(self, container_id: str, action: str, description: str = None,
+                 user_id: str = None, tenant_id: str = None, session_id: str = None):
+        self._container_id = container_id
+        self._action = action
+        self._description = description
+        self._user_id = user_id
+        self._tenant_id = tenant_id
+        self._session_id = session_id
+
+    @property
+    def container_id(self):
+        return self._container_id
+
+    @property
+    def action(self):
+        return self._action
 
+    @property
+    def description(self):
+        return self._description
 
-def init_sql(app: Flask) -> None:
-    from commons.data_access_layer.sql import init_app
-    init_app(app)
+    @property
+    def user_id(self):
+        return self._user_id
 
+    @property
+    def tenant_id(self):
+        return self._tenant_id
 
-def init_cosmos_db(app: Flask) -> None:
-    from commons.data_access_layer.cosmos_db import init_app
-    init_app(app)
+    @property
+    def session_id(self):
+        return self._session_id

commons/data_access_layer/events_model.py

@@ -0,0 +1,6 @@
+from azure.cosmos import PartitionKey
+
+container_definition = {  # pragma: no cover
+    'id': 'event',
+    'partition_key': PartitionKey(path='/tenant_id')
+}

commons/data_access_layer/sql.py

@@ -1,9 +1,7 @@
-from datetime import datetime
-
 from flask import Flask
 from flask_sqlalchemy import SQLAlchemy
 
-from commons.data_access_layer.database import CRUDDao, ID_MAX_LENGTH
+from commons.data_access_layer.database import CRUDDao
 
 db: SQLAlchemy = None
 

commons/git_hooks/enforce_semantic_commit_msg.py

@@ -15,12 +15,10 @@
 
 COMMIT_MSG_REGEX = r'(build|ci|docs|feat|fix|perf|refactor|style|test|chore|revert)(\([\w\-]+\))?:\s.*'
 
-
 # Get the commit message file
 commit_msg_file = open(sys.argv[1])  # The first argument is the file
 commit_msg = commit_msg_file.read()
 
-
 if re.match(COMMIT_MSG_REGEX, commit_msg) is None:
     print(ERROR_MSG)
     sys.exit(1)

migrations/02-add-events.py

@@ -0,0 +1,19 @@
+def up():
+    from commons.data_access_layer.cosmos_db import cosmos_helper
+    import azure.cosmos.exceptions as exceptions
+    from commons.data_access_layer.events_model import container_definition as event_definition
+    from . import app
+
+    app.logger.info("Creating container events...")
+
+    try:
+        app.logger.info('- Event')
+        cosmos_helper.create_container(event_definition)
+    except exceptions.CosmosResourceExistsError as e:
+        app.logger.warning("Unexpected error while creating container for events: %s" % e.message)
+
+    app.logger.info("Done!")
+
+
+def down():
+    print("Not implemented!")

requirements/time_tracker_events/dev.txt

@@ -0,0 +1,5 @@
+# requirements/time_tracker_events/dev.txt
+
+
+# Include the prod resources
+-r prod.txt

requirements/time_tracker_events/prod.txt

@@ -0,0 +1,4 @@
+# requirements/time_tracker_events/prod.txt
+
+# Include the requirements of that folder, required there by convention
+-r ../../time_tracker_events/requirements.txt

setup.cfg

@@ -12,6 +12,9 @@ branch = True
 source =
     time_tracker_api
     commons
+    time_tracker_events
+omit =
+    time_tracker_events/handle_*_events_trigger/*
 
 [report]
 exclude_lines =

setup.py

@@ -1,6 +1,7 @@
-from setuptools import setup, find_packages
 import sys
 
+from setuptools import setup, find_packages
+
 
 def get_version() -> str:
     version = {}