There's currently about a 3 hour delay in starting jobs because the queue is so backed up. Need to start backing off to allow it to catch up.

• Add metrics and traces to Amplify for QoS purposes (otel)
• Add QoS dashboard + alerting
• Add (generic) ability to extra information from jobs and push to Bacalhau dashboard (e.g. types of data)
• Add metrics to Bacalhau dashboard to link Bacalhau jobs to content types (and more)

Interesting one here, parquet doesn't have a registered mime type yet. Wonder if tika can parse?

Metadata | predicate [text/csv| application/parquet] -> Load and produce summaries of data -> Merge

Ideas:

• Data type detected
• https://www.tensorflow.org/tfx/guide/statsgen
• Important stats: https://www.notion.so/pl-strflt/Project-Amplify-Automatic-data-augmentation-ada5cd98838642fd8122d9afb57a3e49?pvs=4#c91b71e9d44c437b8564a800b7e56f62
• https://subsets.io/
• https://framework.frictionlessdata.io/docs/checks/baseline.html

From issue #26

• Add output information to API
• Node ID should be job ID
• Add status information to API
• Change API to allow for proper dags
• Re-add job api
• Swap-out workflows API for nodes API

https://stackoverflow.com/questions/20847674/ffmpeg-libx264-height-not-divisible-by-2

See this error: https://filecoinproject.slack.com/archives/C02RLM3JHUY/p1681807393198339?thread_ts=1681754897.016109&cid=C02RLM3JHUY

• Document thoroughly what predicate is and what is for
• Add usage docs to docs.bacalhau.org

• Add API
• Alter task code so that when no workflow is named it runs that CID over all workflows
  * [ ] Cache/Dedup workflows with same jobs Will be addressed in #27
• Produce single derivative (optional flag)

The logic isn't quite right. Test it better.

Add a step in the merge job that adds a .amplify.lock file and then in the metadata step check for that file. If it finds it, skip.

See the tags build. Just need to add another build container.

• Rate limit the queue
• Rate limit the clients based upon the queue
• Use a prioritised queue so manual submitters can bump their job up the queue
• Record queue performance metrics
• Report predicted delay in API

Not sure why.

Repro:

go run . run QmbRr4kUXMxQfZPnLUSb1kSMDvFtBUcv1HVSDaAUKe4ePj
b get 0b7f6e1a-ef94-441d-9679-ef23904504d0

❯ ll job-0b7f6e1a/default
total 288
-rw-r--r--@ 1 enricorotundo  staff    11K Apr 19 13:10 image1.jpg
-rw-r--r--@ 1 enricorotundo  staff    11K Apr 19 13:10 image1.jpg~
-rw-r--r--@ 1 enricorotundo  staff   4.5K Apr 19 13:10 image2.jpg
-rw-r--r--@ 1 enricorotundo  staff   4.5K Apr 19 13:10 image2.jpg~
-rw-r--r--@ 1 enricorotundo  staff   6.2K Apr 19 13:10 image3.jpg
-rw-r--r--@ 1 enricorotundo  staff   6.2K Apr 19 13:10 image3.jpg~
-rw-r--r--@ 1 enricorotundo  staff   5.0K Apr 19 13:10 image4.jpg
-rw-r--r--@ 1 enricorotundo  staff   5.1K Apr 19 13:10 image4.jpg~
-rw-r--r--@ 1 enricorotundo  staff   9.2K Apr 19 13:10 image4sa.jpg
....

??

Brain dump

• IPFS data statistics workflow
• Format conversion, like csv to parquet, or csv to json
• Validation
• Descriptions

Use Cases

IPFS Data Statistics

Important to do this first, because there's no point working on ${DataType} use cases if there are no ${DataTypes} in IPFS -- {insert data type here}.

Metadata -> Pushing metadata info to external database (security concern, whitelist?)

Web-Focussed Image Compression

Metadata | predicate image/*-> Image Processing -> Merge

Web-Focussed Video Compression

Metadata | predicate video/*-> Video Processing -> Merge

Transcription of Video and Audio

Metadata | predicate [video|audio]/* -> Transcription model(s) -> Merge

CSV/Parquet Summary Statistics

Interesting one here, parquet doesn't have a registered mime type yet. Wonder if tika can parse?

Metadata | predicate [text/csv| application/parquet] -> Load and produce summaries of data -> Merge

${DataType} Enrichment

For example, given CSV file, columns have a lat and long, run job that converts lat/long to country/city and creates output csv with the same row format, then merge back together with original data.

Metadata | predicate ${DataType} -> Parse columnar data type | predicate ${ColumnDataType} -> Data Enrichment -> Merge

Image Dataset Analysis

https://cleanvision.readthedocs.io/

Metadata | predicate image/*-> Image analysis -> Merge

For this CID QmTEvry1uo8qoqBMCSdHobsS7RVXr2M4JZRZXTUKVpoMdp <- a blob video

This execution: bacalhau describe 0cfd3408-7f59-47f9-91e2-739be937affa

This output:

❯ ls -lah job-0cfd3408/default
total 5.2M
drwxr-xr-x 8 phil  256 Apr 20 09:32 .
drwxr-xr-x 7 phil  224 Apr 20 09:32 ..
-rw-r--r-- 1 phil 877K Apr 20 09:32 scaled_1080_video.mp4
-rw-r--r-- 1 phil 877K Apr 20 09:32 scaled_144_video.mp4
-rw-r--r-- 1 phil 877K Apr 20 09:32 scaled_240_video.mp4
-rw-r--r-- 1 phil 877K Apr 20 09:32 scaled_360_video.mp4
-rw-r--r-- 1 phil 877K Apr 20 09:32 scaled_480_video.mp4
-rw-r--r-- 1 phil 877K Apr 20 09:32 scaled_720_video.mp4

Note how they're all the same size.

• Make the canvas bigger
• Add the predicate so you can see when it's stopped
• Add it to the execution page, and color the nodes depending on state

I note that this frictionless also does conversion. That could be cool. Convert all structured data formats into other structured data formats (e.g. csv->xls)

Question: does this command work with any structured type? Looks like you’re just testing with csv.

Yeah, I think that’s worth doing. Could you test adding the https://framework.frictionlessdata.io/docs/console/convert.html command to your run script please? Looks like you have to install specific packages for other data types.

Can you add more functionality?

• Work on adding all data formats that are listed under data formats: https://framework.frictionlessdata.io/docs/formats/html.html
• Add the convert step to convert all formats to all other formats.

• Test current release scripts. Should work (copied from bac) but haven't tested yet.
• Ideally want to do CD
• Figure out where to host the infra (does it run on a separate node?)
• Write deployment scripts
• Hook up GH CI to auto-deploy on release. Yolo, can't be doing with manual deploys.

Why does this job not show the stdout/result even when the job was successful?

http://amplify.bacalhau.org/api/v0/queue/4f101e53-6851-4617-99cb-c60e3ae48ffc

Have the ability to change the log level, but it doesn't work...

• Filecoin deal trigger
• IPFS DHT trigger
• IPFS PubSub trigger
• HTTP watch trigger
• IPFS stream trigger

Should just skip over that input or something, no need to fail subsequent nodes (unless all inputs have failed, obviously)

• Add proper UI and landing page

Phil

• Clearly define job input/output interface
• Change so that all jobs are SingleJob (and refactor)
  • Remove MapJob
  • Refactor
• Remove/Simplify composite
  • Remove children
  • Refactor
• Workers should work on job level i.e. Want to rate limit the number of concurrent bacalhau jobs.

Might be a JavaScript thing. But I did find this which works in go: https://d2lang.com/tour/api

• Release note generator
• Docker Job tester
• #30

Goal: Preserve the input directory structure at the output (e.g. consider /foo/bar.jpg and /hey/bar.jpg)

Implementation:

• python script to shell out the job's command
  • job's stdout/err and exit code should be returned/printed correctly
  • handle case when in file(s) extension is not available (some processors e.g. ffmpeg require that)
  • replace /input/<path> with /outputs/<path>
  • script can take 1 parameter (e.g. resolution for image resize job), it will be used to create "levels" in the output dir (e.g. with param 720p,1080p the output dir structure would be /outputs/720p/<path> and /outputs/1080p/<path>.
  • no external dependencies other than just python3
• write test for this python utils
• containerize the script above
• convert the existing jobs to make use of the container above
• publish container to ghcr.io in this repo

Why not using Bash:

• Escaping hell
• many cases to handle: single file CID vs directory CID, file with or without extension
• Unsupported files (e.g. .DS_Store) shall be handled
• should check for Empty files
• handle uppercase vs lowecase extensions, more complexity
• Not quite testable.
• Manipulating paths as strings is too fragile

• Add a DB to production
• Develop code to persist queue information
• Develop code to persist dag information
• Refactor dag if necessary
• Pagination in DB queries
• Pagination in API

• Add log message when skipping jobs
• Bac describe xxx log message should include job info
  * [ ] Reinstate amplify run job command (possibly?) Not urgent.
• Change default root directory name so as it's not confused with the root user
• Add ability to assume the "default" output when specifying inputs/outputs
• Remove need for specifying path in input. If not required then don't mount.
  * [ ] Amplify create graph command (or equivalent) to plot the DAG Deferred to UI work. Not urgent.
• Show that image on the graph page in the server.
• Print full server address so people can click on it
• Add form to submit job

Is --gpu 1 supported?

Bad for demos/usability to get put to the back of the queue when submitting manually. Should move straight to the front.

E.g. this looks legit: http://amplify.bacalhau.org/api/v0/queue/8ff41b43-98cd-45e9-ad44-32051f8b6e42

StdOut: {"Content-Type":"application/json; charset=ISO-8859-1"} {"Content-Type":"application/json; charset=ISO-8859-1"}

But json isn't reflected in the table...

• #24
• #22
• #23

• Add conditional workflow
  • Need some way of adding the previous stdout to the next stdout so downstream jobs can use previous information
• #31

Metadata | predicate image/*-> Image Processing -> Merge

#26

• rename nodes to graph
• Rename nodes api to graph
• Rename step_id to node_id

• Change the API to use normal pagination according to jsonapi.
• Migrate the datasource to use https://github.com/henvo/ra-jsonapi-client

http://amplify.bacalhau.org/api/v0/queue?order=DESC&page=1&perPage=10&sort=meta.submitted

Test by setting all tests to use a :ro volume.

https://github.com/uber-go/goleak

Since we use goroutines a lot, worth testing for leaks.