Improve repository management

[lamby]

re. Message-ID: <b55d886c-2664-7670-f5d9-11ab64709db3@turnkeylinux.org>

[lamby]

https://debian-handbook.info/browse/stable/sect.setup-apt-package-repository.html
mini-buildd
mini-dinstall
https://github.com/lkorigin/laniakea ← used by purism
https://mirrorer.alioth.debian.org/
aptly

[JedMeister]

To clarify for anyone else listening in, the main aim at this point is to support an authorised person (initially just myself and Alon, but others in the future) being able to upload a binary (deb) file and have it auto added to the main TurnKey repo.

I could be wrong, but I think that Alon may currently be using reprepro locally to create the required repo hierarchy. Then using rsync to push to the (remote) repo. It works, but after having a read about mini-dinstall that seems like it may be a much better option?!

I like the idea of configuring a minimalist user on a remote server, which we could upload (via sftp) to and it would be auto picked up by mini-dinstall and the appropriate files updated and signed. We could then control who has access via authorised SSH keys.

My only concern going that route is that it'd be nice to log who did what and when. Suggestions on how we'd best do that would be warmly welcomed. I guess we could configure a new user for each uploader and either share an upload dir, or perhaps just have an upload dir each and have mini-dinstall moniter them all? Anyway, just throwing some ideas around.

In the longer term, it'd be great to also support auto package building when a repo is updated (e.g. via git hooks). We'd probably want that to only auto push these auto built debs to a testing repo though. The package could be manually migrated to main (via mini-dinstall or similar) once it had been battle tested. Let's leave that part for now though.

[lamby]

(Very quickly; you want to sign packages locally using your own GPG and dput or dupload them (with a .changes file in an upload queue that some script picks up that validates those keys, rather than having a magic dir each. This will give you logging for free too. More soon.)

[lamby]

@JedMeister

So, I had a really good play with the candiates here today. After trying out a few I think we want to use reprepro. It has all the features we want and is really lightweight to boot. In terms of the end UI (ie. you, me, Alon, etc.), what you would do is something like:

$ debsign turnkey /path/to/packagename_1.0-1_amd64.changes

… to sign it with your own, personal, GPG key. You would then upload to the incoming location (set in ~/dput.cf)

$ dput turnkey /path/to/packagename_1.0-1_amd64.changes

… and then once reprepro kicks in (either run via cron or via inotify etc.) it will examine the package and do everything for you and finally signing the Release file with the archive keyring. It determines whether you are allowed to upload there which is based entirely on the signature of the .changes file just like in Debian, and this is how we can restrict access.

It easily supports the use-case of only allowing a subset of folks to upload to various archives or even distributions, depending on how we separate the security and regular archives (where is the security archive? I don't see it on http://archive.turnkeylinux.org/ FYI.

Anyway, let me know how you want to proceed from here. :)

[JedMeister]

@lamby - that sounds awesome mate! I love it! 👍

I'll speak with Alon ASAP and hopefully we'll get his blessing to move forward. I'm not 100% sure on the best way to move forward from here, but hopefully he has some deployment suggestions. Then we'll get you to implement it! Yay! 🎉

PS the security repo is on http://archive.turnkeylinux.org/ although perhaps not where you might expect, it uses a "dist" of 'jessie-security` i.e.:

http://archive.turnkeylinux.org/debian/dists/jessie-security/

[lamby]

Cool - let me know! Fun project :)

(Perfect, so it's one archive for with a separate distribution. This is supported in reprepro.)

[JedMeister]

I spoke with Alon last night and whilst he likes your plan, he has changed his mind on this. For now at least, we're just going to just work around it by he and I sharing the private release key. His main concern was about the risk of the server hosting the private release keys getting compromised and a malicious actor then being able to push packages to our repos.

So unfortunately that means that this project is now on ice. 😢

Apologies on the run around on this...

[lamby]

His main concern was about the risk of the server hosting the private release keys getting compromised

That's completely understandable. However, surely Debian is susceptible to this? :)

[JedMeister]

However, surely Debian is susceptible to this? :)

I did mention that! And Alon acknowledged it. But he also noted that there is a lot more activity on Debian, with multiple security conscious people monitoring things (at least I would assume so!). He argued that would be a (at least somewhat) mitigating factor. Not sure if that's how it is really, but I get his point.

I did suggest that perhaps we could create a specific AWS server for the purpose of uploading (which would then push to the repo proper) and it only be started when required (with everybody with access having an IAMs role with appropriate permissions). That would add an additional layer of security (the server could only be launched by authorised people in the first place). Alon argued that that is probably overkill for our immediate purposes (and he's probably right).

Personally, I'm a big believer in being proactive in developing longer term solutions. I understand that for a small bootstrapped company, focusing on getting the best leverage is important and being as efficient as possible is a good thing. However, I have argued that doing this now would be worth the effort as it will save future efforts and would be easily scaled later.

Regardless, on this occasion I got vetoed. So I'll suck it up and direct my attention to other battles! 😄

Having said that, I'm sure that we'll come back to this...

[lamby]

Presumably a trusted third-party hosting the repository (and key) would not be an avenue to explore either, or..?