Skip to content

ci: Add vnu to dev docker image #3724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
NGPixel merged 2 commits into from
Mar 22, 2022
Merged

ci: Add vnu to dev docker image #3724

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
389ed63
ci: Add vnu to dev docker image
larseggert Mar 22, 2022
e49d58a
Use default-jdk from apt, and just download the vnu JAR blob
larseggert Mar 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 127 additions & 123 deletions dev/docker-test-base/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,123 +1,127 @@
FROM python:3.6-bullseye
LABEL maintainer="IETF Tools Team <tools-discuss@ietf.org>"

ENV DEBIAN_FRONTEND=noninteractive


RUN apt-get update
RUN apt-get -qy upgrade

# Add Node.js Source
RUN curl -fsSL https://deb.nodesource.com/setup_16.x | bash -

# Add Docker Source
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install the packages we need
RUN apt-get update
RUN apt-get install -qy \
apache2-utils \
apt-file \
apt-utils \
bash \
build-essential \
curl \
docker-ce-cli \
enscript \
gawk \
gcc \
ghostscript \
git \
gnupg \
graphviz \
jq \
less \
libcairo2-dev \
libgtk2.0-0 \
libgtk-3-0 \
libnotify-dev \
libgconf-2-4 \
libgbm-dev \
libnss3 \
libxss1 \
libasound2 \
libxtst6 \
libmagic-dev \
libmariadb-dev \
libtidy-dev \
locales \
mariadb-client \
netcat \
nodejs \
pigz \
pv \
python3-ipython \
ripgrep \
rsync \
rsyslog \
ruby \
ruby-rubygems \
unzip \
wget \
xauth \
xvfb \
yang-tools

# Install kramdown-rfc2629 (ruby)
RUN gem install kramdown-rfc2629

# Install chromedriver
COPY docker/scripts/app-install-chromedriver.sh /tmp/app-install-chromedriver.sh
RUN sed -i 's/\r$//' /tmp/app-install-chromedriver.sh && \
chmod +x /tmp/app-install-chromedriver.sh
RUN /tmp/app-install-chromedriver.sh

# Get rid of installation files we don't need in the image, to reduce size
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# "fake" dbus address to prevent errors
# https://github.com/SeleniumHQ/docker-selenium/issues/87
ENV DBUS_SESSION_BUS_ADDRESS=/dev/null

# avoid million NPM install messages
ENV npm_config_loglevel warn
# allow installing when the main user is root
ENV npm_config_unsafe_perm true
# disable NPM funding messages
ENV npm_config_fund false

# Set locale to en_US.UTF-8
RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment && \
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && \
echo "LANG=en_US.UTF-8" > /etc/locale.conf && \
dpkg-reconfigure locales && \
locale-gen en_US.UTF-8 && \
update-locale LC_ALL en_US.UTF-8
ENV LC_ALL en_US.UTF-8

# Install idnits
ADD https://raw.githubusercontent.com/ietf-tools/idnits-mirror/main/idnits /usr/local/bin/
RUN chmod +rx /usr/local/bin/idnits

# Install current datatracker python dependencies
COPY requirements.txt /tmp/pip-tmp/
RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
&& rm -rf /tmp/pip-tmp

# Turn off rsyslog kernel logging (doesn't work in Docker)
RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf

# Fetch wait-for utility
ADD https://raw.githubusercontent.com/eficode/wait-for/v2.1.3/wait-for /usr/local/bin/
RUN chmod +rx /usr/local/bin/wait-for

# Create workspace
RUN mkdir -p /workspace
WORKDIR /workspace

# Install NPM modules
COPY package.json package.json
RUN npm install --no-audit
RUN rm -f package.json package-lock.json
FROM python:3.6-bullseye
LABEL maintainer="IETF Tools Team <tools-discuss@ietf.org>"

ENV DEBIAN_FRONTEND=noninteractive


RUN apt-get update
RUN apt-get -qy upgrade

# Add Node.js Source
RUN curl -fsSL https://deb.nodesource.com/setup_16.x | bash -

# Add Docker Source
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install the packages we need
RUN apt-get update
RUN apt-get install -qy \
apache2-utils \
apt-file \
apt-utils \
bash \
build-essential \
curl \
docker-ce-cli \
default-jdk \
enscript \
gawk \
gcc \
ghostscript \
git \
gnupg \
graphviz \
jq \
less \
libcairo2-dev \
libgtk2.0-0 \
libgtk-3-0 \
libnotify-dev \
libgconf-2-4 \
libgbm-dev \
libnss3 \
libxss1 \
libasound2 \
libxtst6 \
libmagic-dev \
libmariadb-dev \
libtidy-dev \
locales \
mariadb-client \
netcat \
nodejs \
pigz \
pv \
python3-ipython \
ripgrep \
rsync \
rsyslog \
ruby \
ruby-rubygems \
unzip \
wget \
xauth \
xvfb \
yang-tools

# Install kramdown-rfc2629 (ruby)
RUN gem install kramdown-rfc2629

# Install chromedriver
COPY docker/scripts/app-install-chromedriver.sh /tmp/app-install-chromedriver.sh
RUN sed -i 's/\r$//' /tmp/app-install-chromedriver.sh && \
chmod +x /tmp/app-install-chromedriver.sh
RUN /tmp/app-install-chromedriver.sh

# Get rid of installation files we don't need in the image, to reduce size
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# "fake" dbus address to prevent errors
# https://github.com/SeleniumHQ/docker-selenium/issues/87
ENV DBUS_SESSION_BUS_ADDRESS=/dev/null

# avoid million NPM install messages
ENV npm_config_loglevel warn
# allow installing when the main user is root
ENV npm_config_unsafe_perm true
# disable NPM funding messages
ENV npm_config_fund false

# Set locale to en_US.UTF-8
RUN echo "LC_ALL=en_US.UTF-8" >> /etc/environment && \
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && \
echo "LANG=en_US.UTF-8" > /etc/locale.conf && \
dpkg-reconfigure locales && \
locale-gen en_US.UTF-8 && \
update-locale LC_ALL en_US.UTF-8
ENV LC_ALL en_US.UTF-8

# Install idnits
ADD https://raw.githubusercontent.com/ietf-tools/idnits-mirror/main/idnits /usr/local/bin/
RUN chmod +rx /usr/local/bin/idnits

# Install current datatracker python dependencies
COPY requirements.txt /tmp/pip-tmp/
RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
&& rm -rf /tmp/pip-tmp

# Turn off rsyslog kernel logging (doesn't work in Docker)
RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf

# Install the Nu Html Checker (v.Nu)
ADD ADD https://github.com/validator/validator/releases/download/latest/vnu.jar /

# Fetch wait-for utility
ADD https://raw.githubusercontent.com/eficode/wait-for/v2.1.3/wait-for /usr/local/bin/
RUN chmod +rx /usr/local/bin/wait-for

# Create workspace
RUN mkdir -p /workspace
WORKDIR /workspace

# Install NPM modules
COPY package.json package.json
RUN npm install --no-audit
RUN rm -f package.json package-lock.json
6 changes: 2 additions & 4 deletions docker/app.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ RUN apt-get install -qy \
bash \
build-essential \
curl \
default-jdk \
docker-ce-cli \
enscript \
fish \
Expand Down Expand Up @@ -125,10 +126,7 @@ RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf
RUN sed -i 's/#force_color_prompt=/force_color_prompt=/' /root/.bashrc

# Install the Nu Html Checker (v.Nu)
ADD https://github.com/validator/validator/releases/download/latest/vnu.linux.zip /
RUN unzip -d / /vnu.linux.zip
RUN cp -r /vnu-runtime-image/* /usr/local
RUN rm -rf /vnu.linux.zip /vnu-runtime-image
ADD https://github.com/validator/validator/releases/download/latest/vnu.jar /

ADD https://raw.githubusercontent.com/eficode/wait-for/v2.1.3/wait-for /usr/local/bin/
RUN chmod +rx /usr/local/bin/wait-for
Expand Down
2 changes: 2 additions & 0 deletions ietf/utils/test_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -634,6 +634,8 @@ def setup_test_environment(self, **kwargs):
settings.VNU = subprocess.Popen(
[
"java",
"-jar",
"/vnu.jar",
"-Dnu.validator.servlet.bind-address=127.0.0.1",
"nu.validator.servlet.Main",
"8888",
Expand Down
2 changes: 1 addition & 1 deletion ietf/utils/test_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ def get(self, path, *args, skip_verify=False, **extra):
if settings.VNU:
# First, run through https://validator.github.io/validator/
result = subprocess.run(
["java", "nu.validator.client.HttpClient", "-"],
["java", "-jar", "/vnu.jar", "nu.validator.client.HttpClient", "-"],
input=r.content,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
Expand Down