Skip to content

Commit 2b478da

Browse files
rjsparksrjsparks
committed
Merged branch ^/personal/lars/7.39.1.dev0@19465 from lars@eggert.org:
 Rework the docker container. - Legacy-Id: 19507
2 parents 5959779 + 2b80cf7 commit 2b478da

14 files changed

Lines changed: 439 additions & 1135 deletions

File tree

‎docker/Dockerfile‎

Lines changed: 118 additions & 171 deletions
Original file line numberDiff line numberDiff line change
@@ -1,185 +1,132 @@
1-
# -*- shell-mode -*-
2-
# This file is a docker (https://www.docker.com/what-docker) recipe, which can be used to build
3-
# a docker image which is ready to run a datatracker in development mode.
1+
# This is a Dockerfile with everything in it to run the IETF datatracker.
42
#
5-
# It is used to build an image (once you've installed docker) using a command like this (assuming
6-
# suitable replacement of $variables:
3+
# If you make changes to the datatracker that add new dependencies (python
4+
# packages or otherwise), you need to rebuild this image to make them
5+
# available. Do this in the top-level directory of your datatracker source
6+
# tree:
77
#
8-
# $ docker build -t $yourdockerid/datatracker:$version
8+
# docker/build
99
#
10-
# To use a pre-built image, assuming we're on OS X and have a checked-out datatracker repository
11-
# at /Users/$login/src/6.8.1.dev0, you would start (again assuming you've installed docker)
12-
# a container from an image, as follows:
13-
#
14-
# $ docker run -ti --name=$containername -v /Users/$login:/home/$login levkowetz/datatracker:$version /bin/bash
15-
#
16-
# This maps your home directory to /home/$login in the container, and starts it running /bin/bash.
17-
#
18-
# In this first version, the docker environment is set up so that tests will run successfully,
19-
# but the database has *not* been loaded with a dump, and supporting files (drafts, charters, etc.)
20-
# have *not* been downloaded.
21-
22-
FROM dyne/devuan:beowulf
23-
LABEL maintainer="Henrik Levkowetz <henrik@levkowetz.com>"
24-
25-
# Default django runserver port
26-
EXPOSE 8000
27-
28-
# Run apt-get noninteractive
29-
ENV DEBIAN_FRONTEND=noninteractive
30-
ENV DEVUAN_FRONTEND=noninteractive
31-
32-
# Uncomment this to be able to install and run apt-show-versions:
33-
RUN rm -v /etc/apt/apt.conf.d/docker-compress
34-
RUN rm -v /var/lib/apt/lists/*lz4
35-
36-
RUN apt-get update --allow-releaseinfo-change
37-
RUN apt-get install -qy apt-transport-https
38-
39-
# Use backports, updates, and security updates
40-
RUN echo "deb http://deb.devuan.org/merged beowulf main contrib non-free" > /etc/apt/sources.list
41-
RUN echo "deb http://deb.devuan.org/merged beowulf-security main contrib non-free" >> /etc/apt/sources.list
42-
RUN echo "deb http://deb.devuan.org/merged beowulf-updates main contrib non-free" >> /etc/apt/sources.list
43-
RUN echo "deb http://deb.devuan.org/merged beowulf-backports main contrib non-free" >> /etc/apt/sources.list
44-
45-
# Remove some excludes for the docker image
46-
RUN sed -i -e '/^path-exclude=.*\/groff/d' \
47-
-e '/^path-exclude=.*\/locale/d' \
48-
-e '/^path-exclude=.*\/man/d' /etc/dpkg/dpkg.cfg.d/docker-excludes
49-
50-
# Install needed packages
10+
# You can then execute the datatracker like this (also from the top-level
11+
# datatracker source directory):
5112
#
52-
# We're not including graphviz and ghostscript, needed for the 3 document
53-
# dependency graph tests; they would increase the size of the image by about
54-
# 15%, about 100MB.
55-
56-
# Fetch apt package information, and upgrade to latest package versions
57-
58-
RUN apt-get update
59-
RUN apt-get -qy upgrade
60-
61-
# Install the packages we need
62-
RUN apt-get install -qy \
63-
build-essential \
64-
bzip2 \
65-
ca-certificates \
66-
colordiff \
67-
gawk \
68-
gcc \
69-
ipython \
70-
jq \
71-
less \
72-
libbz2-dev \
73-
libdb5.3-dev \
74-
libexpat1-dev \
75-
libffi-dev \
76-
libgdbm-dev \
77-
libjpeg62-turbo-dev \
78-
liblzma-dev \
79-
libmagic1 \
80-
libmariadbclient-dev \
81-
libncurses5-dev \
82-
libncursesw5-dev \
83-
libreadline-dev \
84-
libsqlite3-dev \
85-
libssl-dev \
86-
libsvn1 \
87-
libxml2-dev \
88-
libxslt-dev \
89-
libz-dev \
90-
libffi-dev \
91-
locales \
92-
make \
93-
man \
94-
mariadb-client \
95-
mariadb-server \
96-
openssh-client \
97-
patch \
98-
procps \
99-
pv \
100-
rsync \
101-
rsyslog \
102-
subversion \
103-
sudo \
104-
uuid-dev \
105-
vim \
106-
wget \
107-
xz-utils\
108-
zile \
109-
zlib1g-dev
110-
111-
# Postgresql packages
112-
RUN apt-get install -qy \
113-
postgresql-11 \
114-
postgresql-server-dev-11
115-
116-
# Get the key used to sign the libyang repo
117-
RUN wget -nv http://download.opensuse.org/repositories/home:liberouter/Debian_9.0/Release.key
118-
RUN apt-key add - < Release.key
119-
RUN rm Release.key
120-
121-
# Add apt source entry for libyang
122-
RUN echo "deb http://download.opensuse.org/repositories/home:/liberouter/Debian_9.0/ /" >> /etc/apt/sources.list.d/libyang.list
13+
# docker/run
12314

124-
# Update the package defs, and install the desired mysql from the mysql repo
125-
RUN apt-get update
126-
RUN apt-get install -qy libyang1
15+
FROM ubuntu:hirsute
16+
LABEL maintainer="IETF Tools Team <tools-discuss@ietf.org>"
12717

128-
# This is expected to exist by the mysql startup scripts:
129-
#RUN touch /etc/mysql/debian.cnf
130-
# ------------------------------------------------------------------------------
131-
132-
# Get rid of installation files we don't need in the image, to reduce size
133-
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
134-
135-
# Enable some common locales
136-
RUN sed -i -e 's/^. en_US/en_US/' -e 's/^. en_GB/en_GB/' -e 's/^. en_IE/en_IE/' /etc/locale.gen
137-
RUN locale-gen
138-
139-
# Remove an rsyslog module that we don't need, which also requires extra permissions
140-
RUN sed -i -e '/load="imklog"/d' /etc/rsyslog.conf
141-
142-
# Set up root password
143-
RUN echo "root:root" | chpasswd
18+
# Default django runserver port
19+
EXPOSE 8000
14420

145-
# MySQL
146-
VOLUME /var/lib/mysql
21+
# Default mysqld/mariadb port
22+
EXPOSE 3306
14723

148-
# idnits and dependencies
24+
ENV DEBIAN_FRONTEND=noninteractive
25+
RUN apt-get -y update && \
26+
# apt-get upgrade is normally not a good idea, but this is a dev container
27+
apt-get upgrade && \
28+
# Install all dependencies that are available as packages
29+
apt-get -y install --no-install-recommends \
30+
apache2-utils \
31+
apt-file \
32+
apt-utils \
33+
curl \
34+
enscript \
35+
gcc \
36+
ghostscript \
37+
git \
38+
gnupg \
39+
graphviz \
40+
libmagic-dev \
41+
libmariadb-dev \
42+
locales \
43+
mariadb-server \
44+
npm \
45+
pigz \
46+
pv \
47+
python-is-python3 \
48+
python3-dev \
49+
python3-pip \
50+
rsyslog \
51+
unzip \
52+
yang-tools && \
53+
# Since snap doesn't work in Docker containers, install chromedriver per
54+
# https://gist.github.com/varyonic/dea40abcf3dd891d204ef235c6e8dd79#gistcomment-3160722
55+
curl https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
56+
echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list && \
57+
apt-get update -y && \
58+
apt-get install -y --no-install-recommends google-chrome-stable && \
59+
CHROMEVER=$(google-chrome --product-version | grep -o "[^\.]*\.[^\.]*\.[^\.]*") && \
60+
DRIVERVER=$(curl "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_$CHROMEVER") && \
61+
curl -L -O -C - "http://chromedriver.storage.googleapis.com/$DRIVERVER/chromedriver_linux64.zip" && \
62+
unzip chromedriver_linux64.zip -d /bin && \
63+
rm chromedriver_linux64.zip && \
64+
# Install some other packages that are not dependencies but make life easier
65+
apt-get -y install --no-install-recommends \
66+
fish \
67+
less \
68+
nano \
69+
ripgrep \
70+
zsh && \
71+
# Reduce image footprint (not that it matters given the size of the above)
72+
apt-get -y clean && \
73+
rm -rf /var/lib/apt/lists/*
74+
75+
# Set locale to en_US.UTF-8
76+
RUN dpkg-reconfigure locales && \
77+
locale-gen en_US.UTF-8 && \
78+
update-locale LC_ALL en_US.UTF-8
79+
ENV LC_ALL en_US.UTF-8
80+
81+
# Install bower
82+
RUN npm install -g bower
83+
84+
# Install idnits
14985
ADD https://tools.ietf.org/tools/idnits/idnits /usr/local/bin/
15086
RUN chmod +rx /usr/local/bin/idnits
15187

152-
# Directory for Mac certs
153-
RUN mkdir /etc/certificates
154-
155-
# # Python 3
156-
# # Comment in if OS does not provide python3.6, which is the current
157-
# # production version
158-
ENV PYVER=3.6.10
159-
ENV PYREV=3.6
160-
161-
WORKDIR /usr/src
162-
RUN wget -q https://www.python.org/ftp/python/$PYVER/Python-$PYVER.tar.xz
163-
RUN tar xJf Python-$PYVER.tar.xz
164-
RUN rm Python-$PYVER.tar.xz
165-
WORKDIR /usr/src/Python-$PYVER/
166-
RUN ./configure
167-
RUN make
168-
RUN make altinstall
169-
WORKDIR /usr/src
170-
RUN rm -rf /usr/src/Python-$PYVER/
171-
172-
ENV HOSTNAME="datatracker"
173-
174-
ENV DDIR="/usr/local/share/datatracker"
175-
RUN mkdir -p $DDIR
176-
WORKDIR $DDIR
177-
178-
COPY requirements.txt ./
179-
COPY setprompt ./
180-
88+
# Install current datatracker python dependencies
89+
COPY requirements.txt /
90+
RUN pip install -r /requirements.txt
91+
92+
# Turn off rsyslog kernel logging (doesn't work in Docker)
93+
RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf
94+
95+
# Allow access to mariadb over the network
96+
RUN sed -i 's/127.0.0.1/0.0.0.0/' /etc/mysql/mariadb.conf.d/50-server.cnf
97+
98+
# Turn on mariadb performance_schema
99+
RUN sed -i 's/\[mysqld\]/\[mysqld\]\nperformance_schema=ON/' /etc/mysql/mariadb.conf.d/50-server.cnf
100+
101+
# Make the mariadb sys schema available for possible installation
102+
# We would normally use the next line, but that has a bug:
103+
# ADD https://github.com/FromDual/mariadb-sys/archive/master.zip /
104+
# This is the repo that has the PR:
105+
ADD https://github.com/grooverdan/mariadb-sys/archive/refs/heads/master.zip /
106+
RUN unzip /master.zip
107+
RUN rm /master.zip
108+
109+
# Colorize the bash shell
110+
RUN sed -i 's/#force_color_prompt=/force_color_prompt=/' /root/.bashrc
111+
112+
# Make a database dump available as part of the image, for if a user doesn't
113+
# have one installed locally yet - this saves a bunch of time then
114+
ADD https://www.ietf.org/lib/dt/sprint/ietf_utf8.sql.gz /
115+
RUN pigz -v -d /ietf_utf8.sql.gz && \
116+
sed -i -e 's/ENGINE=MyISAM/ENGINE=InnoDB/' /ietf_utf8.sql
117+
RUN service mariadb start && \
118+
echo "This sequence will take a long time, please be patient" && \
119+
mysqladmin -u root --default-character-set=utf8 create ietf_utf8 && \
120+
bash -c "cd /mariadb-sys-master && mysql --user root < sys_10.sql" && \
121+
bash -c "mysql --user root ietf_utf8 <<< \"GRANT ALL PRIVILEGES ON *.* TO django@localhost IDENTIFIED BY 'RkTkDPFnKpko'; FLUSH PRIVILEGES;\"" && \
122+
bash -c "mysql --user=django --password=RkTkDPFnKpko -f ietf_utf8 < /ietf_utf8.sql" && \
123+
service mariadb stop && \
124+
rm -rf /ietf_utf8.sql /mariadb-sys-master && \
125+
mv /var/lib/mysql /
126+
127+
# Copy the startup file
181128
COPY docker-init.sh /docker-init.sh
182129
RUN chmod +x /docker-init.sh
183-
ENTRYPOINT ["/docker-init.sh"]
184130

185-
CMD /bin/bash
131+
WORKDIR /root/src
132+
ENTRYPOINT ["/docker-init.sh"]

‎docker/README.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Datatracker Development in Docker
2+
3+
## Getting started
4+
5+
1. [Set up Docker](https://docs.docker.com/get-started/) on your preferred
6+
platform.
7+
8+
2. If you have a copy of the datatracker code checked out already, simply `cd`
9+
to the top-level directory.
10+
11+
If not, check out a datatracker branch as usual. We'll check out `trunk`
12+
below, but you can use any branch:
13+
14+
svn co https://svn.ietf.org/svn/tools/ietfdb/trunk
15+
cd trunk
16+
17+
3. **TEMPORARY:** Replace the contents of the `docker` directory with [Lars'
18+
files](https://svn.ietf.org/svn/tools/ietfdb/personal/lars/7.39.1.dev0/docker/).
19+
20+
4. **TEMPORARY:** Until [Lars'
21+
changes](https://svn.ietf.org/svn/tools/ietfdb/personal/lars/7.39.1.dev0/docker/)
22+
have been merged and a docker image is available for download, you will need
23+
to build it locally:
24+
25+
docker/build
26+
27+
This will take a while, but only needs to be done once.
28+
29+
5. Use the `docker/run` script to start the datatracker container. You will be
30+
dropped into a shell from which you can start the datatracker and execute
31+
related commands as usual, for example
32+
33+
ietf/manage.py runserver 0.0.0.0:8000
34+
35+
to start the datatracker.
36+
37+
You can also pass additional arguments to `docker/run`, in which case they
38+
will be executed in the container (instead of a shell being started.)
39+
40+
If you do not already have a copy of the IETF database available in the
41+
`data` directory, one will be downloaded and imported the first time you run
42+
`docker/run`. This will take some time.
43+
44+
Once the datatracker has started, you should be able to open
45+
[http://localhost:8000](http://localhost:8000) in a browser and see the
46+
landing page.
47+
48+
## Troubleshooting
49+
50+
- If the database fails to start, the cause is usually an incompatibility
51+
between the database that last touched the files in `data/mysql` and the
52+
database running inside the docker container.
53+
54+
The solution is to blow away your existing database (`rm -rf data/mysql`). A
55+
fresh copy will be retrieved and imported next time you do `docker/run`, which
56+
should resolve this issue.

0 commit comments

Comments
 (0)