Skip to content

Commit 84b1e70

Browse files
authored
Merge pull request #69 from ioet/feature/refactor-dal-for-cosmos-sql-api#50
Create Data Access Layer for Cosmos SQL API
2 parents 3767cda + 6b17a36 commit 84b1e70

36 files changed

+879
-85
lines changed

.env.template

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
1-
# Package where the app is located
1+
# API
2+
## Package where the app is located
23
export FLASK_APP=time_tracker_api
34

4-
# The database connection URI. Check out the README.md for more details
5-
DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
5+
# Common attributes
6+
## In case you use an Azure SQL database, you must specify the database connection URI. Check out the README.md for more details
7+
#export DATABASE_URI=mssql+pyodbc://<user>:<password>@time-tracker-srv.database.windows.net/<database>?driver\=ODBC Driver 17 for SQL Server
8+
9+
## For Azure Cosmos DB
10+
export DATABASE_ACCOUNT_URI=https://<project_db_name>.documents.azure.com:443
11+
export DATABASE_MASTER_KEY=<db_master_key>
12+
export DATABASE_NAME=<db_name>
13+
### or
14+
# export DATABASE_URI=AccountEndpoint=<ACCOUNT_URI>;AccountKey=<ACCOUNT_KEY>

Dockerfile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ COPY . .
88

99
RUN apk update \
1010
&& apk add --no-cache $buildDeps gcc unixodbc-dev \
11-
&& pip3 install --no-cache-dir -r requirements/prod.txt \
11+
&& pip3 install --no-cache-dir -r requirements/time_tracker_api/prod.txt \
1212
&& curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.5.2.1-1_amd64.apk \
1313
&& curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/mssql-tools_17.5.2.1-1_amd64.apk \
1414
&& curl -O https://download.microsoft.com/download/e/4/e/e4e67866-dffd-428c-aac7-8d28ddafb39b/msodbcsql17_17.5.2.1-1_amd64.sig \
@@ -27,4 +27,4 @@ ENV FLASK_APP time_tracker_api
2727

2828
EXPOSE 5000
2929

30-
CMD ["gunicorn", "-b 0.0.0.0:5000", "run:app"]
30+
CMD ["gunicorn", "-b 0.0.0.0:5000", "api:app"]

README.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# time-tracker-api
22

3-
The API of the TSheets killer app.
3+
This is the mono-repository for the backend services and their common codebase
44

55
## Getting started
66
Follow the following instructions to get the project ready to use ASAP.
@@ -30,10 +30,16 @@ automatically [pip](https://pip.pypa.io/en/stable/) as well.
3030
3131
- Install the requirements:
3232
```
33-
python3 -m pip install -r requirements/<stage>.txt
33+
python3 -m pip install -r requirements/<app>/<stage>.txt
3434
```
3535
36-
The `stage` can be `dev` or `prod`.
36+
Where <app> is one of the executable app namespace, e.g. `time_tracker_api`.
37+
The `stage` can be
38+
39+
* `dev`: Used for working locally
40+
* `prod`: For anything deployed
41+
42+
3743
Remember to do it with Python 3.
3844
3945
@@ -94,7 +100,7 @@ The [integrations tests](https://en.wikipedia.org/wiki/Integration_testing) veri
94100
are working well together. These are the default tests we should run:
95101
96102
```dotenv
97-
python3 -m pytest -v --ignore=tests/sql_repository_test.py
103+
python3 -m pytest -v --ignore=tests/commons/data_access_layer/azure/sql_repository_test.py
98104
```
99105

100106
As you may have noticed we are ignoring the tests related with the repository.

run.py renamed to api.py

File renamed without changes.
File renamed without changes.
File renamed without changes.
Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
import dataclasses
2+
import uuid
3+
from typing import Callable
4+
5+
import azure.cosmos.cosmos_client as cosmos_client
6+
import azure.cosmos.exceptions as exceptions
7+
from azure.cosmos import ContainerProxy
8+
from flask import Flask
9+
10+
11+
class CosmosDBFacade:
12+
def __init__(self, app: Flask): # pragma: no cover
13+
self.app = app
14+
15+
db_uri = app.config.get('DATABASE_URI')
16+
if db_uri is None:
17+
app.logger.warn("DATABASE_URI was not found. Looking for alternative variables.")
18+
account_uri = app.config.get('DATABASE_ACCOUNT_URI')
19+
if account_uri is None:
20+
raise EnvironmentError("DATABASE_ACCOUNT_URI is not defined in the environment")
21+
22+
master_key = app.config.get('DATABASE_MASTER_KEY')
23+
if master_key is None:
24+
raise EnvironmentError("DATABASE_MASTER_KEY is not defined in the environment")
25+
26+
self.client = cosmos_client.CosmosClient(account_uri, {'masterKey': master_key},
27+
user_agent="CosmosDBDotnetQuickstart",
28+
user_agent_overwrite=True)
29+
else:
30+
self.client = cosmos_client.CosmosClient.from_connection_string(db_uri)
31+
32+
db_id = app.config.get('DATABASE_NAME')
33+
if db_id is None:
34+
raise EnvironmentError("DATABASE_NAME is not defined in the environment")
35+
36+
self.db = self.client.get_database_client(db_id)
37+
38+
def create_container(self, container_definition: dict):
39+
try:
40+
return self.db.create_container(**container_definition)
41+
42+
except exceptions.CosmosResourceExistsError: # pragma: no cover
43+
self.app.logger.info('Container with id \'{0}\' was found'.format(container_definition["id"]))
44+
45+
def delete_container(self, container_id: str):
46+
try:
47+
return self.db.delete_container(container_id)
48+
49+
except exceptions.CosmosHttpResponseError: # pragma: no cover
50+
self.app.logger.info('Container with id \'{0}\' was not deleted'.format(container_id))
51+
52+
53+
cosmos_helper: CosmosDBFacade = None
54+
55+
56+
class CosmosDBModel():
57+
def __init__(self, data):
58+
names = set([f.name for f in dataclasses.fields(self)])
59+
for k, v in data.items():
60+
if k in names:
61+
setattr(self, k, v)
62+
63+
64+
class CosmosDBRepository:
65+
def __init__(self, container_id: str,
66+
mapper: Callable = None,
67+
custom_cosmos_helper: CosmosDBFacade = None):
68+
global cosmos_helper
69+
self.cosmos_helper = custom_cosmos_helper or cosmos_helper
70+
if self.cosmos_helper is None: # pragma: no cover
71+
raise ValueError("The cosmos_db module has not been initialized!")
72+
self.mapper = mapper
73+
self.container: ContainerProxy = self.cosmos_helper.db.get_container_client(container_id)
74+
75+
@classmethod
76+
def from_definition(cls, container_definition: dict,
77+
mapper: Callable = None,
78+
custom_cosmos_helper: CosmosDBFacade = None):
79+
return cls(container_definition['id'], mapper, custom_cosmos_helper)
80+
81+
def create(self, data: dict, mapper: Callable = None):
82+
function_mapper = self.get_mapper_or_dict(mapper)
83+
return function_mapper(self.container.create_item(body=data))
84+
85+
def find(self, id: str, partition_key_value, visible_only=True, mapper: Callable = None):
86+
found_item = self.container.read_item(id, partition_key_value)
87+
function_mapper = self.get_mapper_or_dict(mapper)
88+
return function_mapper(self.check_visibility(found_item, visible_only))
89+
90+
def find_all(self, partition_key_value: str, max_count=None, offset=0,
91+
visible_only=True, mapper: Callable = None):
92+
# TODO Use the tenant_id param and change container alias
93+
max_count = self.get_page_size_or(max_count)
94+
result = self.container.query_items(
95+
query="""
96+
SELECT * FROM c WHERE c.tenant_id=@tenant_id AND {visibility_condition}
97+
OFFSET @offset LIMIT @max_count
98+
""".format(visibility_condition=self.create_sql_condition_for_visibility(visible_only)),
99+
parameters=[
100+
{"name": "@tenant_id", "value": partition_key_value},
101+
{"name": "@offset", "value": offset},
102+
{"name": "@max_count", "value": max_count},
103+
],
104+
partition_key=partition_key_value,
105+
max_item_count=max_count)
106+
107+
function_mapper = self.get_mapper_or_dict(mapper)
108+
return list(map(function_mapper, result))
109+
110+
def partial_update(self, id: str, changes: dict, partition_key_value: str,
111+
visible_only=True, mapper: Callable = None):
112+
item_data = self.find(id, partition_key_value, visible_only=visible_only)
113+
item_data.update(changes)
114+
return self.update(id, item_data, mapper=mapper)
115+
116+
def update(self, id: str, item_data: dict, mapper: Callable = None):
117+
function_mapper = self.get_mapper_or_dict(mapper)
118+
return function_mapper(self.container.replace_item(id, body=item_data))
119+
120+
def delete(self, id: str, partition_key_value: str, mapper: Callable = None):
121+
return self.partial_update(id, {
122+
'deleted': str(uuid.uuid4())
123+
}, partition_key_value, visible_only=True, mapper=mapper)
124+
125+
def check_visibility(self, item, throw_not_found_if_deleted):
126+
if throw_not_found_if_deleted and item.get('deleted') is not None:
127+
raise exceptions.CosmosResourceNotFoundError(message='Deleted item',
128+
status_code=404)
129+
130+
return item
131+
132+
def create_sql_condition_for_visibility(self, visible_only: bool, container_name='c') -> str:
133+
if visible_only:
134+
# We are considering that `deleted == null` is not a choice
135+
return 'NOT IS_DEFINED(%s.deleted)' % container_name
136+
return 'true'
137+
138+
def get_mapper_or_dict(self, alternative_mapper: Callable) -> Callable:
139+
return alternative_mapper or self.mapper or dict
140+
141+
def get_page_size_or(self, custom_page_size: int) -> int:
142+
# TODO The default value should be taken from the Azure Feature Manager
143+
# or any other repository for the settings
144+
return custom_page_size or 100
145+
146+
147+
def init_app(app: Flask) -> None:
148+
global cosmos_helper
149+
cosmos_helper = CosmosDBFacade(app)

time_tracker_api/sql_repository.py renamed to commons/data_access_layer/sql.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ def handle_commit_issues(f):
1414
def rollback_if_necessary(*args, **kw):
1515
try:
1616
return f(*args, **kw)
17-
except:
17+
except: # pragma: no cover
1818
db.session.rollback()
1919
raise
2020

@@ -92,7 +92,7 @@ def delete(self, id):
9292
self.repository.remove(id)
9393

9494

95-
class SQLSeeder(Seeder):
95+
class SQLSeeder(Seeder): # pragma: no cover
9696
def run(self):
9797
print("Provisioning database...")
9898
db.create_all()

requirements/azure_cosmos.txt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# requirements/azure_cosmos.txt
2+
3+
# For Cosmos DB
4+
5+
# Azure Cosmos DB official library
6+
azure-core==1.1.1
7+
azure-cosmos==4.0.0b6
8+
certifi==2019.11.28
9+
chardet==3.0.4
10+
idna==2.8
11+
six==1.13.0
12+
urllib3==1.25.7
13+
virtualenv==16.7.9
14+
virtualenv-clone==0.5.3

requirements/commons.txt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# requirements/commons.txt
2+
3+
# For Common dependencies
4+
5+
# Handling requests
6+
requests==2.23.0
7+
8+
# To create sample content in tests and API documentation
9+
Faker==4.0.2

0 commit comments

Comments
 (0)