Skip to content

Commit 6b17a36

Browse files
author
EliuX
committed
Create and test Cosmos db repository #50
1 parent 0bb0371 commit 6b17a36

File tree

28 files changed

+850
-78
lines changed

28 files changed

+850
-78
lines changed

.env.template

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,13 @@
22
## Package where the app is located
33
export FLASK_APP=time_tracker_api
44

5-
65
# Common attributes
7-
## The database connection URI. Check out the README.md for more details
8-
DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
6+
## In case you use an Azure SQL database, you must specify the database connection URI. Check out the README.md for more details
7+
#export DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
8+
9+
## For Azure Cosmos DB
10+
export DATABASE_ACCOUNT_URI=https://<project_db_name>.documents.azure.com:443
11+
export DATABASE_MASTER_KEY=<db_master_key>
12+
export DATABASE_NAME=<db_name>
13+
### or
14+
# export DATABASE_URI=AccountEndpoint=<ACCOUNT_URI>;AccountKey=<ACCOUNT_KEY>

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ COPY . .
88

99
RUN apk update \
1010
&& apk add --no-cache $buildDeps gcc unixodbc-dev \
11-
&& pip3 install --no-cache-dir -r requirements/prod.txt \
11+
&& pip3 install --no-cache-dir -r requirements/time_tracker_api/prod.txt \
1212
&& curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.5.2.1-1_amd64.apk \
1313
&& curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.5.2.1-1_amd64.apk \
1414
&& curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.5.2.1-1_amd64.sig \

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# time-tracker-api
22

3-
This is the mono-repository for the backend services and common codebase
3+
This is the mono-repository for the backend services and their common codebase
44

55
## Getting started
66
Follow the following instructions to get the project ready to use ASAP.
@@ -100,7 +100,7 @@ The [integrations tests](https://en.wikipedia.org/wiki/Integration_testing) veri
100100
are working well together. These are the default tests we should run:
101101
102102
```dotenv
103-
python3 -m pytest -v --ignore=tests/sql_repository_test.py
103+
python3 -m pytest -v --ignore=tests/commons/data_access_layer/azure/sql_repository_test.py
104104
```
105105

106106
As you may have noticed we are ignoring the tests related with the repository.

commons/data_access_layer/azure/__init__.py

Whitespace-only changes.
Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
import dataclasses
2+
import uuid
3+
from typing import Callable
4+
5+
import azure.cosmos.cosmos_client as cosmos_client
6+
import azure.cosmos.exceptions as exceptions
7+
from azure.cosmos import ContainerProxy
8+
from flask import Flask
9+
10+
11+
class CosmosDBFacade:
12+
def __init__(self, app: Flask): # pragma: no cover
13+
self.app = app
14+
15+
db_uri = app.config.get('DATABASE_URI')
16+
if db_uri is None:
17+
app.logger.warn("DATABASE_URI was not found. Looking for alternative variables.")
18+
account_uri = app.config.get('DATABASE_ACCOUNT_URI')
19+
if account_uri is None:
20+
raise EnvironmentError("DATABASE_ACCOUNT_URI is not defined in the environment")
21+
22+
master_key = app.config.get('DATABASE_MASTER_KEY')
23+
if master_key is None:
24+
raise EnvironmentError("DATABASE_MASTER_KEY is not defined in the environment")
25+
26+
self.client = cosmos_client.CosmosClient(account_uri, {'masterKey': master_key},
27+
user_agent="CosmosDBDotnetQuickstart",
28+
user_agent_overwrite=True)
29+
else:
30+
self.client = cosmos_client.CosmosClient.from_connection_string(db_uri)
31+
32+
db_id = app.config.get('DATABASE_NAME')
33+
if db_id is None:
34+
raise EnvironmentError("DATABASE_NAME is not defined in the environment")
35+
36+
self.db = self.client.get_database_client(db_id)
37+
38+
def create_container(self, container_definition: dict):
39+
try:
40+
return self.db.create_container(**container_definition)
41+
42+
except exceptions.CosmosResourceExistsError: # pragma: no cover
43+
self.app.logger.info('Container with id \'{0}\' was found'.format(container_definition["id"]))
44+
45+
def delete_container(self, container_id: str):
46+
try:
47+
return self.db.delete_container(container_id)
48+
49+
except exceptions.CosmosHttpResponseError: # pragma: no cover
50+
self.app.logger.info('Container with id \'{0}\' was not deleted'.format(container_id))
51+
52+
53+
cosmos_helper: CosmosDBFacade = None
54+
55+
56+
class CosmosDBModel():
57+
def __init__(self, data):
58+
names = set([f.name for f in dataclasses.fields(self)])
59+
for k, v in data.items():
60+
if k in names:
61+
setattr(self, k, v)
62+
63+
64+
class CosmosDBRepository:
65+
def __init__(self, container_id: str,
66+
mapper: Callable = None,
67+
custom_cosmos_helper: CosmosDBFacade = None):
68+
global cosmos_helper
69+
self.cosmos_helper = custom_cosmos_helper or cosmos_helper
70+
if self.cosmos_helper is None: # pragma: no cover
71+
raise ValueError("The cosmos_db module has not been initialized!")
72+
self.mapper = mapper
73+
self.container: ContainerProxy = self.cosmos_helper.db.get_container_client(container_id)
74+
75+
@classmethod
76+
def from_definition(cls, container_definition: dict,
77+
mapper: Callable = None,
78+
custom_cosmos_helper: CosmosDBFacade = None):
79+
return cls(container_definition['id'], mapper, custom_cosmos_helper)
80+
81+
def create(self, data: dict, mapper: Callable = None):
82+
function_mapper = self.get_mapper_or_dict(mapper)
83+
return function_mapper(self.container.create_item(body=data))
84+
85+
def find(self, id: str, partition_key_value, visible_only=True, mapper: Callable = None):
86+
found_item = self.container.read_item(id, partition_key_value)
87+
function_mapper = self.get_mapper_or_dict(mapper)
88+
return function_mapper(self.check_visibility(found_item, visible_only))
89+
90+
def find_all(self, partition_key_value: str, max_count=None, offset=0,
91+
visible_only=True, mapper: Callable = None):
92+
# TODO Use the tenant_id param and change container alias
93+
max_count = self.get_page_size_or(max_count)
94+
result = self.container.query_items(
95+
query="""
96+
SELECT * FROM c WHERE c.tenant_id=@tenant_id AND {visibility_condition}
97+
OFFSET @offset LIMIT @max_count
98+
""".format(visibility_condition=self.create_sql_condition_for_visibility(visible_only)),
99+
parameters=[
100+
{"name": "@tenant_id", "value": partition_key_value},
101+
{"name": "@offset", "value": offset},
102+
{"name": "@max_count", "value": max_count},
103+
],
104+
partition_key=partition_key_value,
105+
max_item_count=max_count)
106+
107+
function_mapper = self.get_mapper_or_dict(mapper)
108+
return list(map(function_mapper, result))
109+
110+
def partial_update(self, id: str, changes: dict, partition_key_value: str,
111+
visible_only=True, mapper: Callable = None):
112+
item_data = self.find(id, partition_key_value, visible_only=visible_only)
113+
item_data.update(changes)
114+
return self.update(id, item_data, mapper=mapper)
115+
116+
def update(self, id: str, item_data: dict, mapper: Callable = None):
117+
function_mapper = self.get_mapper_or_dict(mapper)
118+
return function_mapper(self.container.replace_item(id, body=item_data))
119+
120+
def delete(self, id: str, partition_key_value: str, mapper: Callable = None):
121+
return self.partial_update(id, {
122+
'deleted': str(uuid.uuid4())
123+
}, partition_key_value, visible_only=True, mapper=mapper)
124+
125+
def check_visibility(self, item, throw_not_found_if_deleted):
126+
if throw_not_found_if_deleted and item.get('deleted') is not None:
127+
raise exceptions.CosmosResourceNotFoundError(message='Deleted item',
128+
status_code=404)
129+
130+
return item
131+
132+
def create_sql_condition_for_visibility(self, visible_only: bool, container_name='c') -> str:
133+
if visible_only:
134+
# We are considering that `deleted == null` is not a choice
135+
return 'NOT IS_DEFINED(%s.deleted)' % container_name
136+
return 'true'
137+
138+
def get_mapper_or_dict(self, alternative_mapper: Callable) -> Callable:
139+
return alternative_mapper or self.mapper or dict
140+
141+
def get_page_size_or(self, custom_page_size: int) -> int:
142+
# TODO The default value should be taken from the Azure Feature Manager
143+
# or any other repository for the settings
144+
return custom_page_size or 100
145+
146+
147+
def init_app(app: Flask) -> None:
148+
global cosmos_helper
149+
cosmos_helper = CosmosDBFacade(app)

commons/data_access_layer/azure/sql_repository.py renamed to commons/data_access_layer/sql.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ def handle_commit_issues(f):
1414
def rollback_if_necessary(*args, **kw):
1515
try:
1616
return f(*args, **kw)
17-
except:
17+
except: # pragma: no cover
1818
db.session.rollback()
1919
raise
2020

@@ -92,7 +92,7 @@ def delete(self, id):
9292
self.repository.remove(id)
9393

9494

95-
class SQLSeeder(Seeder):
95+
class SQLSeeder(Seeder): # pragma: no cover
9696
def run(self):
9797
print("Provisioning database...")
9898
db.create_all()

requirements/azure_cosmos.txt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# requirements/azure_cosmos.txt
2+
3+
# For Cosmos DB
4+
5+
# Azure Cosmos DB official library
6+
azure-core==1.1.1
7+
azure-cosmos==4.0.0b6
8+
certifi==2019.11.28
9+
chardet==3.0.4
10+
idna==2.8
11+
six==1.13.0
12+
urllib3==1.25.7
13+
virtualenv==16.7.9
14+
virtualenv-clone==0.5.3

requirements/commons.txt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# requirements/commons.txt
2+
3+
# For Common dependencies
4+
5+
# Handling requests
6+
requests==2.23.0
7+
8+
# To create sample content in tests and API documentation
9+
Faker==4.0.2

requirements/sql_db_serverless.txt renamed to requirements/sql_db.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
# requirements/sql_db_serverless.txt
1+
# requirements/sql_db.txt
22

3-
# For SQL database serverless (MS SQL)
3+
# For SQL database (MS SQL)
44

55

66
# SQL Server driver

requirements/time_tracker_api/dev.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@ pytest==5.2.0
1010

1111
# Mocking
1212
pytest-mock==2.0.0
13-
Faker==4.0.2
1413

1514
# Coverage
1615
coverage==4.5.1

0 commit comments

Comments
 (0)