Skip to content

Commit 14ad250

Browse files
dantownsenddantownsend
authored
Schema visualise (piccolo-orm#225)
* basic prototype * render data from apps * allow the user to specify which apps to output * make direction optional * more aliases * added docs * fix lgtm warning * be able to write output to file * don't print out database version error if the proper version can't be retrieved * use warnings instead of print, to prevent polluting output when piping to clipboard or file * prevent more stdout pollution * add example image * add tests * use a random filename for the test * tweak test - mocks have a slightly different API on Python 3.7
1 parent 2048697 commit 14ad250

File tree

9 files changed

+292
-27
lines changed

9 files changed

+292
-27
lines changed

‎docs/src/piccolo/projects_and_apps/images/schema_graph_output.png‎

124 KB
Loading

‎docs/src/piccolo/projects_and_apps/included_apps.rst‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@ Auto includes
99

1010
The following are registered with your :ref:`AppRegistry<AppRegistry>` automatically:
1111

12+
-------------------------------------------------------------------------------
13+
1214
app
1315
~~~
1416

@@ -18,6 +20,8 @@ Lets you create new Piccolo apps. See :ref:`PiccoloApps`.
1820
1921
piccolo app new
2022
23+
-------------------------------------------------------------------------------
24+
2125
asgi
2226
~~~~
2327

@@ -27,6 +31,8 @@ Lets you scaffold an ASGI web app. See :ref:`ASGICommand`.
2731
2832
piccolo asgi new
2933
34+
-------------------------------------------------------------------------------
35+
3036
meta
3137
~~~~
3238

@@ -36,11 +42,15 @@ Tells you which version of Piccolo is installed.
3642
3743
piccolo meta version
3844
45+
-------------------------------------------------------------------------------
46+
3947
migrations
4048
~~~~~~~~~~
4149

4250
Lets you create and run migrations. See :ref:`Migrations`.
4351

52+
-------------------------------------------------------------------------------
53+
4454
playground
4555
~~~~~~~~~~
4656

@@ -51,6 +61,8 @@ Lets you learn the Piccolo query syntax, using an example schema. See
5161
5262
piccolo playground run
5363
64+
-------------------------------------------------------------------------------
65+
5466
project
5567
~~~~~~~
5668

@@ -62,9 +74,14 @@ Lets you create a new ``piccolo_conf.py`` file. See :ref:`PiccoloProjects`.
6274
6375
.. _SchemaApp:
6476

77+
-------------------------------------------------------------------------------
78+
6579
schema
6680
~~~~~~
6781

82+
generate
83+
^^^^^^^^
84+
6885
Lets you auto generate Piccolo ``Table`` classes from an existing database.
6986
Make sure the credentials in ``piccolo_conf.py`` are for the database you're
7087
interested in, then run the following:
@@ -77,6 +94,34 @@ interested in, then run the following:
7794
current form it will save you a lot of time. Make sure you check the
7895
generated code to make sure it's correct.
7996

97+
graph
98+
^^^^^
99+
100+
A basic schema visualisation tool. It prints out the contents of a GraphViz dot
101+
file representing your schema.
102+
103+
.. code-block:: bash
104+
105+
piccolo schema graph
106+
107+
You can pipe the output to your clipboard (``piccolo schema graph | pbcopy``
108+
on a Mac), then paste it into a `website like this <https://dreampuf.github.io/GraphvizOnline>`_
109+
to turn it into an image file.
110+
111+
Or if you have `Graphviz <https://graphviz.org/download/>`_ installed on your
112+
machine, you can do this to create an image file:
113+
114+
.. code-block:: bash
115+
116+
piccolo schema graph | dot -Tpdf -o graph.pdf
117+
118+
Here's an example of a generated image:
119+
120+
.. image:: ./images/schema_graph_output.png
121+
:target: /_images/schema_graph_output.png
122+
123+
-------------------------------------------------------------------------------
124+
80125
shell
81126
~~~~~
82127

@@ -87,6 +132,8 @@ Launches an iPython shell, and automatically imports all of your registered
87132
88133
piccolo shell run
89134
135+
-------------------------------------------------------------------------------
136+
90137
sql_shell
91138
~~~~~~~~~
92139

@@ -101,6 +148,8 @@ need to run raw SQL queries on your database.
101148
For it to work, the underlying command needs to be on the path (i.e. ``psql``
102149
or ``sqlite3`` depending on which you're using).
103150

151+
-------------------------------------------------------------------------------
152+
104153
.. _TesterApp:
105154

106155
tester

‎piccolo/apps/schema/commands/graph.py‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
"""
2+
Credit to the Django Extensions team for inspiring this tool.
3+
"""
4+
5+
import dataclasses
6+
import os
7+
import sys
8+
import typing as t
9+
10+
import jinja2
11+
12+
from piccolo.conf.apps import Finder
13+
14+
TEMPLATE_DIRECTORY = os.path.join(
15+
os.path.dirname(os.path.abspath(__file__)), "templates"
16+
)
17+
18+
JINJA_ENV = jinja2.Environment(
19+
loader=jinja2.FileSystemLoader(searchpath=TEMPLATE_DIRECTORY),
20+
)
21+
22+
23+
@dataclasses.dataclass
24+
class GraphColumn:
25+
name: str
26+
type: str
27+
28+
29+
@dataclasses.dataclass
30+
class GraphTable:
31+
name: str
32+
columns: t.List[GraphColumn]
33+
34+
35+
@dataclasses.dataclass
36+
class GraphRelation:
37+
table_a: str
38+
table_b: str
39+
label: str
40+
41+
42+
def render_template(**kwargs):
43+
template = JINJA_ENV.get_template("graphviz.dot.jinja")
44+
return template.render(**kwargs)
45+
46+
47+
def graph(
48+
apps: str = "all", direction: str = "LR", output: t.Optional[str] = None
49+
):
50+
"""
51+
Prints out a graphviz .dot file for your schema.
52+
53+
:param apps:
54+
The name of the apps to include. If 'all' is given then every app is
55+
included. To specify multiple app names, separate them with commas.
56+
For example --apps="app1,app2".
57+
:param direction:
58+
How the tables should be orientated - by default it's "LR" which is
59+
left to right, so the graph will be landscape. The alternative is
60+
"TB", which is top to bottom, so the graph will be portrait.
61+
:param output:
62+
If specified, rather than printing out the file contents, they'll be
63+
written to this file. For example --output=graph.dot
64+
65+
"""
66+
finder = Finder()
67+
app_names = finder.get_sorted_app_names()
68+
69+
if apps != "all":
70+
given_app_names = [i.strip() for i in apps.split(",")]
71+
delta = set(given_app_names) - set(app_names)
72+
if delta:
73+
sys.exit(f"These apps aren't recognised: {', '.join(delta)}.")
74+
app_names = given_app_names
75+
76+
tables: t.List[GraphTable] = []
77+
relations: t.List[GraphRelation] = []
78+
79+
for app_name in app_names:
80+
app_config = finder.get_app_config(app_name=app_name)
81+
for table_class in app_config.table_classes:
82+
tables.append(
83+
GraphTable(
84+
name=table_class.__name__,
85+
columns=[
86+
GraphColumn(
87+
name=i._meta.name, type=i.__class__.__name__
88+
)
89+
for i in table_class._meta.columns
90+
],
91+
)
92+
)
93+
for fk_column in table_class._meta.foreign_key_columns:
94+
reference_table_class = (
95+
fk_column._foreign_key_meta.resolved_references
96+
)
97+
relations.append(
98+
GraphRelation(
99+
table_a=table_class.__name__,
100+
table_b=reference_table_class.__name__,
101+
label=fk_column._meta.name,
102+
)
103+
)
104+
105+
contents = render_template(
106+
tables=tables, relations=relations, direction=direction
107+
)
108+
109+
if output is None:
110+
print(contents)
111+
else:
112+
with open(output, "w") as f:
113+
f.write(contents)

‎piccolo/apps/schema/commands/templates/graphviz.dot.jinja‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
digraph model_graph {
2+
fontname = "Roboto"
3+
fontsize = 8
4+
splines = true
5+
rankdir = "{{ direction }}";
6+
7+
node [
8+
fontname = "Roboto"
9+
fontsize = 8
10+
shape = "plaintext"
11+
]
12+
13+
edge [
14+
fontname = "Roboto"
15+
fontsize = 8
16+
]
17+
18+
// Tables
19+
{% for table in tables %}
20+
TABLE_{{ table.name }} [label=<
21+
<TABLE BGCOLOR="white" BORDER="1" CELLBORDER="0" CELLSPACING="0">
22+
<TR>
23+
<TD COLSPAN="2" CELLPADDING="5" ALIGN="CENTER" BGCOLOR="#4C89C8">
24+
<FONT FACE="Roboto" COLOR="white" POINT-SIZE="10">
25+
<B>{{ table.name }}</B>
26+
</FONT>
27+
</TD>
28+
</TR>
29+
30+
{% for column in table.columns %}
31+
<TR>
32+
<TD ALIGN="LEFT" BORDER="0">
33+
<FONT FACE="Roboto">
34+
<B>{{ column.name }}</B>
35+
</FONT>
36+
</TD>
37+
<TD ALIGN="LEFT">
38+
<FONT FACE="Roboto">
39+
<B>{{ column.type }}</B>
40+
</FONT>
41+
</TD>
42+
</TR>
43+
{% endfor %}
44+
</TABLE>
45+
>]
46+
{% endfor %}
47+
48+
// Relations
49+
{% for relation in relations %}
50+
TABLE_{{ relation.table_a }} -> TABLE_{{ relation.table_b }}
51+
[label="{{ relation.label }}"] [arrowhead=none, arrowtail=dot, dir=both];
52+
{% endfor %}
53+
}

‎piccolo/apps/schema/piccolo_app.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,16 @@
11
from piccolo.conf.apps import AppConfig, Command
22

33
from .commands.generate import generate
4+
from .commands.graph import graph
45

56
APP_CONFIG = AppConfig(
67
app_name="schema",
78
migrations_folder_path="",
8-
commands=[Command(callable=generate, aliases=["g", "create", "new"])],
9+
commands=[
10+
Command(callable=generate, aliases=["gen", "create", "new"]),
11+
Command(
12+
callable=graph,
13+
aliases=["map", "visualise", "vizualise", "viz", "vis"],
14+
),
15+
],
916
)

‎piccolo/engine/base.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ async def check_version(self):
6969

7070
engine_type = self.engine_type.capitalize()
7171
logger.info(f"Running {engine_type} version {version_number}")
72-
if version_number < self.min_version_number:
72+
if version_number and (version_number < self.min_version_number):
7373
message = (
7474
f"This version of {self.engine_type} isn't supported "
7575
f"(< {self.min_version_number}) - some features might not be "

‎piccolo/engine/postgres.py‎

Lines changed: 9 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
from piccolo.querystring import QueryString
1111
from piccolo.utils.lazy_loader import LazyLoader
1212
from piccolo.utils.sync import run_sync
13-
from piccolo.utils.warnings import Level, colored_string, colored_warning
13+
from piccolo.utils.warnings import Level, colored_warning
1414

1515
asyncpg = LazyLoader("asyncpg", globals(), "asyncpg")
1616

@@ -274,8 +274,7 @@ async def get_version(self) -> float:
274274
except ConnectionRefusedError as exception:
275275
# Suppressing the exception, otherwise importing piccolo_conf.py
276276
# containing an engine will raise an ImportError.
277-
colored_warning("Unable to connect to database")
278-
print(exception)
277+
colored_warning(f"Unable to connect to database - {exception}")
279278
return 0.0
280279
else:
281280
version_string = response[0]["server_version"]
@@ -290,15 +289,13 @@ async def prep_database(self):
290289
f'CREATE EXTENSION IF NOT EXISTS "{extension}"',
291290
)
292291
except asyncpg.exceptions.InsufficientPrivilegeError:
293-
print(
294-
colored_string(
295-
f"=> Unable to create {extension} extension - some "
296-
"functionality may not behave as expected. Make sure "
297-
"your database user has permission to create "
298-
"extensions, or add it manually using "
299-
f'`CREATE EXTENSION "{extension}";`',
300-
level=Level.medium,
301-
)
292+
colored_warning(
293+
f"=> Unable to create {extension} extension - some "
294+
"functionality may not behave as expected. Make sure "
295+
"your database user has permission to create "
296+
"extensions, or add it manually using "
297+
f'`CREATE EXTENSION "{extension}";`',
298+
level=Level.medium,
302299
)
303300

304301
###########################################################################

‎piccolo/main.py‎

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
from piccolo.apps.user.piccolo_app import APP_CONFIG as user_config
2525
from piccolo.conf.apps import AppRegistry, Finder
2626
from piccolo.utils.sync import run_sync
27-
from piccolo.utils.warnings import Level, colored_string
27+
from piccolo.utils.warnings import Level, colored_warning
2828

2929
DIAGNOSE_FLAG = "--diagnose"
3030

@@ -117,18 +117,17 @@ def main():
117117
if havent_ran_count == 1
118118
else f"{havent_ran_count} migrations haven't"
119119
)
120-
print(
121-
colored_string(
122-
message=(
123-
"=> {} been run - the app "
124-
"might not behave as expected.\n"
125-
"To check which use:\n"
126-
" piccolo migrations check\n"
127-
"To run all migrations:\n"
128-
" piccolo migrations forwards all\n"
129-
).format(message),
130-
level=Level.high,
131-
)
120+
121+
colored_warning(
122+
message=(
123+
"=> {} been run - the app "
124+
"might not behave as expected.\n"
125+
"To check which use:\n"
126+
" piccolo migrations check\n"
127+
"To run all migrations:\n"
128+
" piccolo migrations forwards all\n"
129+
).format(message),
130+
level=Level.high,
132131
)
133132
except Exception:
134133
pass

0 commit comments

Comments
 (0)