New appliance - Syncthing

If there is interest.... I will start work on a syncthing appliance. I should be able to get this done quickly.

https://syncthing.net

It is a useful tool for keeping files in sync and I did not see it in the Appliance Candidates.

[JedMeister]

Hey @mptmg - that looks like an awesome appliance candidate and we'd love to add it to the library. However, if I were you, I'd be inclined to hold off for a week or 2.

The rationale behind that is that we're currently pushing out the last of the v14.2 release and I'm madly trying to finalise that. So, I don't plan to add any more new appliances (for v14.2). Although there is one other than has been hanging around for a little while (not as long as yours - but longer than I like), which I might squeeze in if I can.

As soon as that's done, we'll be upgrading the base infrastructure to the new Debian release (Stretch). Once that's done, we'll then turn around and update all the current appliances to v15.0. I can then add any new v15.0 appliances you or others produce then.

If you want, you could make a start before then, but it's highly unlikely that we'll publish a v14.2 release. Having v14.2 based code would still give you a head start for v15.0, but will most likely need some tweaking.

So it's your call, but I think it would be best to wait.

I will wait. I will lay the groundwork for it now but will wait. I wasn't trying to slip in more work for you before 14.2, I just saw that you had integrated Mumble and will have ZoneMinder done shortly as well, so I figured I would tackle another appliance I find useful.

[JedMeister]

No worries. I don't want to put you off, I just wanted to be clear on where we're up to so you have a bit of context.

Mostly, I really just don't want to repeat the drawn out process that has happened with your 2 other submissions. It's a bit embarrassing really...! 😄

We love your contributions, despite the fact that it took us so long to release them. Actually part of the hold up has been that we've been refactoring things behind the scenes to try to do more rapid releases. I'm not yet convinced that it will make the releases tons quicker, but hopefully it should improve things...

Went ahead and finished the work on this and committed it to GitHub. Simple appliance to configure considering it has very few dependencies and inithooks are limited. FWIW here is the link to the repository https://github.com/MPTMG/Syncthing. I will do the artwork shortly. If anything else is needed let me know.

[JedMeister]

Awesome, thanks @mptmg. We're still way behind on our next release, although the pace is starting to pick up. We still have a few major issues that need to be resolved before we can build a release candidate of Core (which will likely be the first public release of v15.0).

So unfortunately, I'm not sure when we'll get to release your new app, but thanks tons for your work.

I should post some instructions on how to start building for Stretch/v15.0 somewhere. It's actually fairly simple...

Builds fine and works on TKL-dev 15.0 rc1

[JedMeister]

Great work @mptmg

I've just had a quick look at the code and here is some feedback/things we'll need to do prior to inclusion. I don't necessarily mean that you need to do them, but they will need to be done before we can add it and we're pretty snowed under ATM, so if you could have a look would be awesome!:

• conf script:

  • good practice to include the -y switch when installing packages in conf scripts. Even if it doesn't need it ATM, future versions may require additional dependencies and without the -y switch the build will stall, waiting for confirmation. So it's a good habit to be in. 😄

  • package(s) installed from third party apt repo need to be pinned (for security). For example see Mediaserver emby.pref - apt preferences overlay file for Emby

  • additionally, the apt repo should be secured via (new practice introduced for v15.0):

    • upstream apt gpg key should be saved in overlay/usr/share/keyrings/APP_NAME.gpg, i.e. overlay/usr/share/keyrings/syncthing.gpg
    • include signed-by line (pointing to gpg key) in the sources.list file. E.g. emby.list overlay file
    • rational and further info, see Fix Apt/GPG for third party repos #1033 - strictly speaking these files don't necessarily need to be in overlay (could be in conf script), but most others are, so it's good for consistency.

• changelog - remove and add single 15.0 entry (appliances should only have entries relating to their release - note that this is initial release).

• readme - adjust to bring formatting and layout into line with other appliance readmes (we semi-automatically process them to produce the appliance page text on the website)

• appliance logo - it will need it's own logo/icon (currently still has Core logo/icon). The logo/icon template and some instructions can be found here: https://github.com/turnkeylinux/artwork

• landing page (i.e. tklweb-cp) or redirect (via webserver, e.g. nginx) - as it's served on a non-standard port, IMO it would be of value to have either a landing page (can be handy to also provide doc links and third party app links) or redirect vanilla http/https (i.e. port 80/443) to the WebUI. TBH, my preference would be the former, but the latter is ok too.

• screenshot(s) - Ideally each appliance should include 3 screenshots. But if there aren't 3 screenshots of value, we can "recycle" common ones (such as Webmin) to make up the 3. I'd delete all 3 currently included as they'll need to be updated for v15.0 anyway. IMO at least one screenshot of the SyncThing webUI shoudl be included, perhaps 2 (or more) if they look interesting. If we include a landing page, then a screenshot of that would also be appropriate. Suggested workflow is noted in the artwork repo readme - more detailed notes/instructions are in the docs.

I will complete the list. Only thing that seems remotely complex is pinning the third party apt repo, but I will look at what was done for emby and try to reproduce for syncthing. Good work on the 15.0-rc1.

[JedMeister]

Thanks mate. Apologies that I didn't give this specific feedback sooner. Between support and trying to polish up v15.0, I've been snowed under...

Ideally, we probably should write up something like a "new appliance checklist" or similar.

If you need any further assistance or guidance, get stuck, etc, please don't hesitate to ask.

Also, there's no super rush on this, just let me know when you're ready. I hope to start releasing v15.0 appliances within the next week, but same as v14.2, they'll be done in batches of ~20 so there'll be about 5 batches, a week or 2 apart. So there's plenty of time. And even if it isn't ready by the last batch, it's not that big a deal. Moving forward, I hope to do more regular builds of v15.x.

I took care of the conf script -y, added the gpg key via overlay, repo via overlay, and pinned the repo.

If I blank out the change log it gives me a build error. What does it reference in the changelog for the build process? If I copy the default changelog from dev or core it builds fine. I have 100% confirmed it has something to do with the changelog. I was not under the impression the changelog could tamper with the build process.

I will update the readme and the art tomorrow.

I do not see the value in including a web server with redirect or tklweb-cp (control panel) as I could easily just change the default hosting port on Syncthing to 80 or 443 and avoid the unnecessary packages. However, if someone is using Syncthing (bothered to read the docs) they will be expecting the webgui on 8384 so changing the default port for the Syncthing webgui would prove confusing for those users. Let me know if you would like the default port changed for the Syncthing webgui, but installing a web server nginx or lighttpd or apache along with php to essentially serve links via one php page to webmin/syncthing/terminal is not something I think has value to the end user. For the experienced user it would involve additional work to close the unnecessary open ports for the web server. You have full permission to give me hell if a ton of users are bothering you for support because they can't type :8384 for the syncthing webgui.

Will post code tomorrow after art and readme is done.

[JedMeister]

Hi @mptmg - Great work.

Apologies if I wasn't clear. I didn't mean remove the changelog completely, I meant just have a single v15.0 entry.

The changelog needs to exist and it also needs to comply with Debian changelog policy. One way to ensure that it does is to copy/paste (or edit) an existing changelog entry and update it as appropriate.

Here's an example changelog for a new appliance called new-appliance:

turnkey-new-appliance-15.0 (1) turnkey; urgency=low

  * Initial appliance release based on LAMP appliance

  * Includes v1.0 of new-appliance, installed from upstream git repo

 -- Jeremy Davis <jeremy@turnkeylinux.org>  Mon, 02 Jul 2018 10:15:36 +1000

IMO a better way is to use the Debian developer dch tool to create a new entry (that's in the devscripts package). I hope to one day make a smaller and easier to use tool for our purposes, but for now you can use it like this:

First set up your TKLDev with the appropriate env vars, here's mine (output of cat /root/.bashrc.d/tkldev):

export DEBFULLNAME="Jeremy Davis"
export DEBEMAIL="jeremy@turnkeylinux.org"

If you want the time/date to be in your local format (rather than UTC) set that up too (i.e. dpkg-reconfigure tzdata).

Then assuming that the changelog doesn't yet exist, within the appliance base directory (e.g. /turnkey/fab/products/new-appliance). Obviously adjust APP_NAME as appropriate:

APP_NAME=turnkey-new-appliance-15.0
dch --create --changelog changelog --package $APP_NAME --newversion 1 --force-distribution turnkey --urgency low

Which will produce this changelog:

turnkey-new-appliance-15.0 (1) UNRELEASED; urgency=low

  * turnkey

 -- Jeremy Davis <jeremy@turnkeylinux.org>  Mon, 02 Jul 2018 10:31:41 +1000

Then edit with your favourite editor and change UNRELEASED to turnkey and replace the turnkey entry with your desired entries.

FWIW, here's how to add a new entry to an existing changelog:

APP_NAME=turnkey-existing-appliance-15.0
dch --changelog changelog --release-heuristic log --package $APP_NAME --newversion 1 --force-distribution turnkey --urgency low

Apologies that I hadn't noted that somewhere previously. We should probably include that in the TKLDev docs...

Instructions were clear that the changelog had to be updated to reflect a new appliance, I just wasn't aware an improperly formatted changelog could cause problems to the build process and hence just blanked it out temporarily, while making other changes. I won't lie after making the minor changes for third party repo when it started throwing a build error I was completely baffled as to what the problem was considering I could not fathom it was the changelog.

Thank you for the detailed instructions on how to properly generate a changelog as it will be useful going forward.