Code Monkey home page Code Monkey logo

yolo's Introduction


Yolo - The Berty Project

Yolo is an over-the-air installation distributor for your mobile applications

Made by Berty Technologies discord github twitter go.dev reference GitHub release

Multi-platform over-the-air installation aggregator (a TestFlight alternative).

Philosophy

Mobile app development workflows should be as free as possible from corporate ecosystem constraints.

Introduction

Yolo is one-stop realtime feed of ready-to-install releases and tests for your apps. We're using it at berty to make releases and test branches ready to download and use on a developer's device within minutes after passing our CI.

We created Yolo to implement the critical features missing from Apple's TestFlight.

Need TestFlight Yolo
Generate artifacts for one build for multiple platforms (e.g iOS, Android) Write manual custom scripts Default CI tool aggregator
Writing documentation Write instructions for each platform One universal doc
Accessing artifacts across platforms Use a separate tool per OS Single aggregated feed
Speedy certificate validation Run from scratch on eachpublication (~30 minutes) In-house certificate and signature (~1 minute)
Generate artifacts at any stage of deployment Stable releases only Deeply customizable: Merges to master, pending pull requests, specific tags... anything that passes the CI!

Getting started

Official Berty's instance

Nothing to install, using a browser, you can list and download the last builds of applications of the Berty ecosystem.

You just need to have credentials provided by the Berty team and go to https://yolo.berty.io.

Note that all the available credentials won't give you access to the same application set.

Call the official API

Using the same credentials you can make direct calls to the API, available here: https://yolo.berty.io/api/.

The API is described in protobuf-format here: https://github.com/berty/yolo/blob/master/api/yolopb.proto.

# last build for the berty app for iOS
curl -su :TOKEN "https://yolo.berty.io/api/build-list?project_id=https://github.com/berty/berty&artifact_kinds=1" | jq '.builds[0]'
{
  "id": "https://buildkite.com/berty/berty-open/builds/535",
  "created_at": "2020-04-29T15:06:33.796Z",
  "state": "Passed",
  "message": "feat: add multipeer connectivity Transport and add it in libp2p\n\nfeat: add multipeer connectivity for mobile devices\n\nfix: fix linux compilation failed\n\nchore: remove old references to BLE\n\nchore: remove xcode project directory\n\nchore: goimports passed\n\nfeat: pass functional logger to the mc transport\n\nchore: improve log message in the mc driver",
  "started_at": "2020-04-29T15:45:14Z",
  "finished_at": "2020-04-29T15:58:41Z",
  "branch": "D4ryl00:feat/multipeer-connectivity-integration",
  "driver": "Buildkite",
  "short_id": "535",
  "has_artifacts": [
    {
      "id": "buildkite_524ced1e072c6bb74e3bf9556854b339",
      "created_at": "2020-04-29T15:06:33.796Z",
      "file_size": "38838861",
      "local_path": "Berty-Yolo-08a8bb0dee9935ab14e62648c6969cd5dfd9f517.ipa",
      "download_url": "https://api.buildkite.com/v2/organizations/berty/pipelines/berty-open/builds/535/jobs/323605e5-72fd-4495-8198-615a68672148/artifacts/16bab990-66ee-4ed5-a9d4-db69704bc0fd/download",
      "mime_type": "application/octet-stream",
      "state": "Finished",
      "kind": "IPA",
      "driver": "Buildkite",
      "has_build_id": "https://buildkite.com/berty/berty-open/builds/535",
      "dl_artifact_signed_url": "/api/artifact-dl/buildkite_524ced1e072c6bb74e3bf9556854b339?sign=REDACTED",
      "plist_signed_url": "%2Fapi%2Fplist-gen%2Fbuildkite_524ced1e072c6bb74e3bf9556854b339.plist%3Fsign%3DREDACTED"
    }
  ],
  "has_commit_id": "08a8bb0dee9935ab14e62648c6969cd5dfd9f517",
  "has_project": {
    "id": "https://github.com/berty/berty",
    "created_at": "2018-07-16T05:21:19Z",
    "updated_at": "2020-04-29T13:13:05Z",
    "driver": "GitHub",
    "name": "berty",
    "description": "Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network",
    "has_owner": {
      "id": "https://github.com/berty",
      "name": "berty",
      "driver": "GitHub",
      "avatar_url": "https://avatars1.githubusercontent.com/u/22157871?v=4",
      "kind": "Organization"
    },
    "has_owner_id": "https://github.com/berty"
  },
  "has_project_id": "https://github.com/berty/berty",
  "has_mergerequest": {
    "id": "https://github.com/berty/berty/pull/1908",
    "created_at": "2020-04-23T08:19:28Z",
    "updated_at": "2020-04-30T09:54:59Z",
    "title": "WIP feat: add the multipeer connectivity transport",
    "message": "Add the multipeer connectivity transport of berty v1 to the master branch of berty\r\n* [x] add the transport + driver in an internal package\r\n* [ ] switch on/off that transport from the front",
    "driver": "GitHub",
    "branch": "D4ryl00:feat/multipeer-connectivity-integration",
    "state": "Opened",
    "commit_url": "https://github.com/berty/berty/commit/5face40d919f102d9d0f2b19061bae666f4b940a",
    "short_id": "1908",
    "has_project": {
      "id": "https://github.com/berty/berty",
      "created_at": "2018-07-16T05:21:19Z",
      "updated_at": "2020-04-29T13:13:05Z",
      "driver": "GitHub",
      "name": "berty",
      "description": "Berty is a secure peer-to-peer messaging app that works with or without internet access, cellular data or trust in the network",
      "has_owner_id": "https://github.com/berty"
    },
    "has_project_id": "https://github.com/berty/berty",
    "has_author": {
      "id": "https://github.com/D4ryl00",
      "name": "D4ryl00",
      "driver": "GitHub",
      "avatar_url": "https://avatars3.githubusercontent.com/u/13605410?v=4",
      "kind": "User"
    },
    "has_author_id": "https://github.com/D4ryl00",
    "has_commit_id": "08a8bb0dee9935ab14e62648c6969cd5dfd9f517"
  },
  "has_mergerequest_id": "https://github.com/berty/berty/pull/1908"
}

Run your own instance of Yolo

You will need to get multiple CI tokens to allow Yolo to fetch your last builds.

Install

go get -u berty.tech/yolo/v2/go/cmd/yolo

Or grab the last Docker image available on the official Docker Registry.

You can start from the official deployment configuration available in ./deployments/yolo.berty.io.

CLI usage

$ yolo -h
USAGE
  server [flags] <subcommand>

SUBCOMMANDS
  server        Start a Yolo Server
  dump-objects
  info

FLAGS
  -v false  increase log verbosity
$ yolo server -h
USAGE
  server

FLAGS
  -auth-salt ...             salt used to generate authentication tokens at the end of the URLs
  -basic-auth-password ...   if set, enables basic authentication
  -bintray-token ...         Bintray API Token
  -bintray-username ...      Bintray username
  -buildkite-token ...       BuildKite API Token
  -circleci-token ...        CircleCI API Token
  -cors-allowed-origins ...  CORS allowed origins (*.domain.tld)
  -db-path :temp:            DB Store path
  -dev-mode false            enable insecure helpers
  -github-token ...          GitHub API Token
  -grpc-bind :9000           gRPC bind address
  -http-bind :8000           HTTP bind address
  -max-builds 100            maximum builds to fetch from external services (pagination)
  -realm Yolo                authentication Realm
  -request-timeout 5s        request timeout
  -shutdown-timeout 6s       server shutdown timeout
  -with-cache false          enable API caching

Troubleshooting

(please use issues)

Development

[TODO]

Architecture

[TODO]

Other Resources

  • UI style and asset specifications on HackMD here
  • Sketch templates here

Contributing

Contribute to Berty

🚧 This beta release is currently configured for our internal projects, but we welcome you to fork our repository to customize it for your own use.

If you want to help out, please see CONTRIBUTING.md.

This repository falls under the Berty Code of Conduct.

You can contact us on the #dev-💻 channel on discord.

License

Dual-licensed under Apache 2.0 and MIT terms.

See the COPYRIGHT file for more details.

yolo's People

Contributors

90dy avatar aeddi avatar alexsland avatar clegirar avatar codefactor-io[bot] avatar doozers avatar dzalevski avatar ekelen avatar fgm avatar gfanton avatar glouvigny avatar imgbotapp avatar jorropo avatar moul avatar moul-bot avatar semgrep-bot avatar sfroment avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

yolo's Issues

Improve UI<->HTTP Responses mapping

To reconcile the mockup and the fields I get from server HTTP response, here are some fields I am missing or unclear about. Would be helpful for me and for others using the API to have them clarified.

Screenshot 2020-04-17 at 12 40 37

  1. ID in build title (circled in red)
  2. ID next to Platform icon+name (circled in red)
  3. Link that 'Logs' icon should point to
  4. Link that Github logo shoudl point to
  5. Icon to use in card header (Pull vs Merge icon next to build title; it's the green merge icon in screenshot)
  6. Source of github user avatar
  7. Author
  8. Possible states for artifacts – is this complete/correct? Finished == Green, Yellow == Building, Pink == Failed
  9. Date/time deltas in artifact list:
  • Calendar: (now - ?)
  • Clock: (now - ?)

🚧 More to come...

  1. Possible states for builds – same question: Passed == Green, Pink == Failed, (Yellow?)
  2. Platform to Kind mappings for Artifacts – IPA/DMG == Apple logo; APK == Android?

[bug] sometimes some builds are missing

I need to investigate, but my current feeling is that there is a trick with the "last_updated"/"since" variables used when calling the CI's API

Bug experienced by @n0izn0iz and @gfanton, I fixed it by trashing the whole database and relaunching the service from scratch

Create Figma Mockups

We will try two new things:

  1. To use Figma instead of Sketch
  2. To delegate completely the styling implementation to someone else

Can you start mockups on Figma of some screens you want to see on yolo?
I've no fear to add more views later, so focus on what you think to be the most useful things for now 👍

Should we initialize React transition as a PWA?

I don't have experience making them, but maybe someone who does has an opinion. If this will primarily be used in web browsers, optimizing from the start for that format could be smart.

For now I will plan to use vanilla responsive React, but anyone's input is welcome.

Related to #12

Prod broken and not detected by the CI

https://yolo.berty.io/ returns a 404 page

this website is not hosted on netlify, but runs the bertytech/yolo docker image which is directly built from the Dockerfile

Here are the build logs when I try to build it myself with docker build -t bertytech/yolo .:

[...]
Step 5/23 : COPY            ./web ./
 ---> cbef17128ce4
Step 6/23 : RUN             npm run build
 ---> Running in 39c749868e86

> [email protected] prebuild /app
> npm run clean-dist


> [email protected] clean-dist /app
> npm run remove-dist && mkdir dist


> [email protected] remove-dist /app
> rimraf ./dist


> [email protected] build /app
> env YOLO_APP_PW=${YOLO_APP_PW} babel-node tools/build.js

Generating minified bundle. This will take a moment...
./node_modules/tabler/scss/tabler.scss
Module build failed (from ./node_modules/mini-css-extract-plugin/dist/loader.js):
ModuleBuildError: Module build failed (from ./node_modules/sass-loader/dist/cjs.js):
SassError: Undefined variable: "$form-check-input-checked-bg-position".
        on line 54 of node_modules/tabler/scss/ui/forms/_form-imagecheck.scss
        from line 26 of node_modules/tabler/scss/_tabler-core.scss
        from line 8 of /app/node_modules/tabler/scss/tabler.scss
>>       background-position: $form-check-input-checked-bg-position;

   ---------------------------^

    at runLoaders (/app/node_modules/webpack/lib/NormalModule.js:316:20)
    at /app/node_modules/loader-runner/lib/LoaderRunner.js:367:11
    at /app/node_modules/loader-runner/lib/LoaderRunner.js:233:18
    at context.callback (/app/node_modules/loader-runner/lib/LoaderRunner.js:111:13)
    at Object.render [as callback] (/app/node_modules/sass-loader/dist/index.js:73:7)
    at Object.done [as callback] (/app/node_modules/neo-async/async.js:8067:18)
    at options.error (/app/node_modules/node-sass/lib/index.js:294:32)
 @ ./src/App.js 8:0-33
 @ ./src/index.js
Removing intermediate container 39c749868e86
 ---> 97441b37106b
Step 7/23 : FROM            golang:1.14-alpine as go-build
 ---> 51e47ee4db58
[...]

As you can see, the npm run build generates an error that results in an empty dist folder, but the exit code of npm run build is 0


Please, make sure that a failing build returns an error instead of failing silently, and then, you can try to fix the build

smart authentication

most of the calls need to be authenticated using BASIC auth, jwt, cookie etc

but some calls are executed from a sandbox, so we will need to generate links containing specific tokens in order to allow a sandboxed client to download an asset without double auth

Add more metadata in the generated .plist file

Web: Add tooltips to icons for desktop

Some icons/components are not intuitive (reason for grayed out filter options, log icon, current filters in header, etc); these should have tooltips.

Use CSS coherently throughout front

Right now it's a mix of .scss global files (_variables.scss) and unscoped local .scss files (Header.scss), and inline styles in JSX. The naming conventions for classes also vary between files.

So, we need to:

  • Name custom (i.e. non-vendor) classes consistently throughout web codebase
  • Remove all inline styles except calculated color based on theme preference
  • All components should use same convention to import CSS, whether with 1-1 stylesheets (Header.js, Header.scss) or something else

Define a format to attach custom metadata with releases

The idea is to create a file format that will be uploaded alongisde the .apk, .ipa etc to add some metadata (branch, author, version, builddate, etc) that can be used to enhance the UI with more info or to create a new API call allowing the apps to automatically find "more recent builds" automatically

Allow the user to configure/override the API token

I just configured a demo on Netlify (https://yolo-demo.netlify.com/)

This demo will have at least to goals:

  • easier testing of the PRs
  • demo

Today, the only way of passing an optional token is by env var when building/running the node webserver, we should also support a way of specifying at runtime, i.e.:

by passing an optional query string or by prompting the user for a token as soon as the JS client detects a 401 reply from the API

Handle server requests for multiple artifact kinds

Currently the golang server only accepts requests for builds with artifacts from only one specified platform (iOS/'1' or android/'2') (or at least that is my understanding of req *yolopb.BuildList_Request; I'm tagging this issue as question for this reason).

Expected behavior: From the designs below, I'm assuming we want to also enable passing multiple platforms in parameters.

image

image

Refactor workers to be more « real-time »

Currently, the batches are long, blocking, and cannot trigger cross-driver events

The idea is to switch to a solution that allows having real workers that process data in real time, update the db in real time, and trigger cross actions

[META] Ideas 💡

Front Improvements

  • Open running builds by default (#308)
  • see the branch that triggered the build
  • support nice URLs instead of complex query strings
  • Add download statistics to build info on front (#305)
  • Cadd a widget that shows running builds (#144, #288)
  • Add merge request sharable link icon functionality to buildcards with MR (#254)
  • Use alternative logo/logo position for small screens (#284)
  • Header in logo resizes bizarrely on small-medium screen (#250)
  • Change link icon --> copy icon (UI) (#259)
  • Make UI button/icon alignment and sizing consistent (#257)
  • Add UI branch filter functionality (#253)
  • Add more views (TV Mode, Dashboard, ...) (#26)
  • Scroll to feed/feed start on feed display toggle

Back Improvements

  • TestFlight integration
  • Android store integration
  • F-droid store integration
  • add direct url to know if there is a new version and return a json + another one to download “latest version” and append them with hash in the generated plist before resigning the binary
  • add a way to force the refresh of the API (caching issue)
  • direct url to dl last artifact -> Home Screen button and in-app update
  • Share apple shortcut
  • support uploading an artifact
  • put/get on s3 instead of local disk to become stateless
  • bigger pagination on build list
  • check if possible to match PRs without match
  • sort by commit date (#458)
  • add a call that can be generated by yolo to give direct download link to "more recent version", and add this link in the generated plist
  • support auto domain expanding(#454)
  • add a .github/yolo.yml config file that configures yolo for this repo, i.e., enabled workflows, etc
  • Define a format to attach custom metadata with releases (#35)
  • github driver: compute artifact duration
  • pagination
  • check if missing a beforeUpdate gorm hook or if there is another reason why the yoloid sometimes disappear
  • support filter: include_wip (#299)
  • support filter: include_closed (#299)
  • support filter: artifact_driver=... (#126)
  • support filter: branch_name=... (#126)
  • support filter: artifact_state=... (#126)
  • support filter: job_name= (#126)
  • support filter: merged VS PR VS other (#126)
  • support filter: job_id= (#126)
  • support input search field instead of checkboxes (#126)
  • support <none> or <other> filter value for filtering booleans (#126)
  • support filter: with_no_mergerequest=true (#399)
  • reprocess existing entities (#298)
  • add a mode that paginate from the creation of the repo
  • store/distribute artifacts with IPFS
  • resign .ipa
  • (re)sign .apk
  • add missing relationship during aggregation: build<->mergerequest on multiple drivers (#146)
  • add missing relationship during aggregation: has_project on multiple drivers (#146)
  • add missing relationship during aggregation: duration field (#146)
  • add missing relationship during aggregation: fetch github commits (#146)
  • add missing relationship during aggregation: fetch github releases (#146)
  • Add Travis driver (#14)
  • Return full URLs (with domain) from the API (#86)
  • webhook to trigger refresh (#89)
  • add a routine that periodically check that existing data in db is still valid (not 404/403) (#235)
  • add a mode that fetches all the build history and also backup all the artifacts locally (#234)
  • Use human-friendly values in query params (#185)
  • create a first "badge" generator (#148)
  • support .exe, .zip, .dmg, .pkg, docker images, etc (#147)
  • yolo can comment pull requests with direct install links (#135)
  • Validate query parameters in API (#115)
  • patch the .ipa, and .apk to add a yolo_update_url dynamically (#41)
  • add a mergerequest_list route or sort/group query param (#2 (comment))
  • add alternative .ipa configuration (various download buttons)
  • if icon cannot be extracted from the artifact, use the org/user image instead of yolo.png
  • consider using the event observer design pattern for aggregators
  • basic SEO and thumbnail sharing with light API that works without token
  • generate a ribbon overlay with beta/PR ID
  • Speed up signature (https://github.com/zhlynn/zsign#how-to-sign-quickly)
  • method to list what artifact should be GCed
  • use pinata/s3 to store artifacts for a long time
  • Investigate why the link is ugly here -> https://yolo.berty.io/?build_id=https://github.com/berty/berty/actions/runs/528752453 + berty/berty#3070

Code Quality

  • Split out messy global state manager (#306)
  • Move inline React-CSS color styles to (s)css module file (#256)
  • Use berty/berty similar coding style to define flags in the go main(#456)
  • Make sure docker build is only installing node production dependencies (#66)
  • Use CSS coherently throughout front (#68)
  • Stop using babel-node for web production build (#70)
  • Handle React component failures gracefully (#72)

Doc

  • Add demo.gif in the README.md (#350)
  • Add a section about "contributing to the front without running your own backend" -> can be moved in the CONTRIBUTING.md file instead (#149)

Testing

  • Create an example repo with different CIs set up (#8)

Refactor schemas

By implementing the bintray driver, I noticed that most of other solutions are very simple:

  • « repos that have builds that have artifacts »
  • a build can be a PR, master, etc

On bintray we have more layers:

  • « project that have versions that have files »
  • a project is not necessarily a repo
  • a version may have files uploaded in different builds

Basically they can all fit in the same schemas, but it can be a good thing to abstract them in a way that allows more flexibility, I.e the « GitHub repo relationship X CI’s project » relationship

Depends on #16

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.