esell / deb-simple Goto Github PK

If you do not want to build from source you can just download a pre-built binary from the Releases section.
Uhm, there is no release. Could you make one?

As this project is about a debian package server, is there a reason why there is none for this project? Could you create one?

Add some documentation around how you would add your repo into the distro's sources.

Currently only testing for the "stable" distro so a bug was missed (#10) where createPackagesGz() was only using the default value. Create another test for non-default distros.

Would be nice to support auto creation of Let's Encrypt certs

As I am no longer using deb-simple in my day-to-day life, it has been mostly ignored as you can likely see from the issues/PRs. If you are interested in taking over and becoming the maintainer of deb-simple, please let me know

Hi,

I would like to mention I created a golang package to debianize files and directories. It is not completely finished but it is already functional.

https://github.com/xor-gate/debpkg

Let me know what you think and maybe you can point me to the right direction or contribute if you would like to (when there is a use-case).

It seems that Bionic Beaver (18.04) requires more fields than Xenial (16.04) in the Packages.gz file...
This worked on Xenial:

Filename: dists/xenial/main/binary-amd64/<name>_<version>_amd64.deb
Size: <size>
MD5sum: <md5>
SHA1: <sha>
SHA256: <sha>

But does not work on Bionic; I added these lines to the extracted file at /var/lib/apt/lists/:

Package: <name>
Architecture: amd64
Version: <version>
Priority: standard
Description: <description>

And it worked fine.

This may be an issue with my debian packaging tool, not necessarily an issue with deb-simple, but just thought you might want to know.

Most times we already have a directory full of packages that we'd like to have served via deb-simple. Is there an easy way to add a config flag that would allow this dir to be served instead of iterating over every file in the dir and POSTing it to the server (which takes a long time when the files are too many or too large).

Happy to take on the work if you think this is functionality you agree with (and have thoughts on how to implement).

Is there a way to use this in a global production environment by blocking add/delete requests?

Right now everything is put into the "stable" distro and the "main" section. Would be nice to be able to change these defaults.

Right now uploads are stored in memory before being written out to disk. If you had a bunch of uploads going for large packages this could potentially be an issue.

Might be worthwhile to research other possible methods/options for uploading files.

We've run into an issue using deb-simple where new packages can't be installed until a couple of minutes after uploading them. We have quite a few version of ~50Mb packages, so it takes a little while to hash them all to produce the Packages file.

One potential fix for this would be to move back to the pre-fsnotify behaviour and rebuild the Packages file as part of the package upload handler. This would mean our CI build wouldn't count as complete until the package was ready to install, which makes sense.

I can understand that being able to drop files into the packages directory is useful behaviour for some users, so I don't want to remove it.

@esell what do you think about making the behaviour here configurable? A setting to control if either the directory should be watched, or the Packages file should be rebuilt as a blocking part of the upload.

I've tried to build debian package in docker container golang:1.12-buster and hit an error

root@3da6c33c12a8:/go/src/github.com/esell/deb-simple# make build-deb
which -s dpkg-deb || { echo "dpkg-deb does not exist, exiting..."; exit 1; }
Illegal option -s
Usage: /usr/bin/which [-a] args
dpkg-deb does not exist, exiting...
make: *** [Makefile:26: build-deb] Error 1

here is my patch to fix it

index 3976ed9..548c147 100644
--- a/Makefile
+++ b/Makefile
@@ -23,8 +23,7 @@ build-linux:
 build-osx:
 	GOOS=darwin go build -o release/$(BINARY)-$(VERSION)-osx
 build-deb:
-	which -s dpkg-deb || { echo "dpkg-deb does not exist, exiting..."; exit 1; }
-	mkdir release/$(BINARY)-$(VERSION)
+	dpkg-deb --version >/dev/null || { echo "dpkg-deb does not exist, exiting..."; exit 1; }
 	mkdir -p release/$(BINARY)-$(VERSION)/usr/local/bin
 	mkdir -p release/$(BINARY)-$(VERSION)/etc/deb-simple
 	cp -r DEBIAN release/$(BINARY)-$(VERSION)/

In func inspectPackageControl , we return the args via return controlBuf.String(), nil. However, I've encountered control files that have a trailing newline. The return string from this func is used to create the 'Packages.gz' file, and if the return string has newlines, these make their way into the Packages.gz file but apt clients (esp. Ubuntu 12.04) can get unhappy about stray newlines.

The fix is to trim trailing blank chars from the controlBuf.String()

Hi,

I currently looking into deb-simple as it looks promising. Unfortunately I encounter a problem that I can't circumvent.

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/localhost:9090_dists_stable_main_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.

my conf.json looks like the default one

{
    "listenPort" : "9090",
    "rootRepoPath" : "/opt/mirror",
    "supportedArch" : ["amd64"],
    "distroNames" : ["stable"],
    "sections" : ["main"],
    "enableSSL" : true,
    "SSLcert" : "server.crt",
    "SSLkey" : "server.key",
    "enableAPIKeys" : false,
    "enableSigning" : true,
    "privateKey" : "./private.key"
}

I generated the private/public key with -k -ke -kn option. The SSLcert is just a snakeoil from my ubuntu.

After initial start I upload some test files from my current deb mirror via for file in $(find . -name *_amd64.deb); do curl -k -X POST 'https://localhost:9090/upload?arch=amd64&distro=stable&section=main' -F "file=@$file"; done. I don't get an error message on upload from curl, nor from the server output (started with -v flag).

My /etc/apt/sources.list contains deb [arch=amd64] https://localhost:9090/ stable main.

How to prevent the error from happening? I already tried sudo rm /var/lib/apt/lists/* -vf without success.

When integrating with other golang software. deb-simple could benefit from an API client package. E.g:

package main

import (
    "github.com/esell/deb-simple/client"
)

func main() {
    client.New(&client.Config {
     APIKey: "1337",
    })
    client.PublishDebFile("myfile.deb")
}

This is a great project, thanks for making it!

Have you considered adding support for using this as a CLI tool? (Or would you accept a PR adding this mode?)

What I mean by that is I would love to be able to run ./deb-simple batch <dir full of deb files> and have it generate all the Packages, Release, GPG-signed files, etc. I would then host the generated files on some static hosting, e.g. S3, GitHub Pages, etc.

In practice, I would run it as part of the CI pipeline for one of my projects, after goreleaser generates the deb files. goreleaser doesn't plan on doing this work itself in the near future: goreleaser/goreleaser#912