Code Monkey home page Code Monkey logo

assets.ubuntu.com's Introduction

assets.ubuntu.com codebase

The assets server manages the assets (mainly images and PDFs) for different Canonical and Ubuntu websites.

It has 2 main parts:

Summary

Creating assets using the web interface

You will need to login with an SSO account that is in the canonical-content-people team.

You can then create assets using the web interface at https://assets.ubuntu.com/manager/create.

Transforming images

When getting an image asset, the asset can be transformed using the op (operation) option.

You can manually specify one of these operations along with their corresponding options:

  • region:
    • rect: The region as x,y,w,h; x,y: top-left position, w,h: width/height of region
  • resize:
    • w: Width
    • h: Height
    • max-width
  • max-height:
  • rotate:
    • deg: Degrees to rotate the image

The default option is resize and can be used without setting op.

The default option is resize and can be used without setting op, e.g.:

https://assets.ubuntu.com/v1/4d7a830e-logo-ubuntuone.png?w=30

Or you can use another feature, like region e.g.:

https://assets.ubuntu.com/v1/4d7a830e-logo-ubuntuone.png?op=region&rect=0,0,50,50

Using the RestAPI

Creating a new asset can you be done using the assets manager, however in case of advanced option such as image transformation or creating redirects, you can use the API directly.

Authentication

The API uses a token based authentication system. You can specify your token in 3 different ways:

  • As a token query parameter: e.g. https://assets.ubuntu.com/v1?token=1234
  • As an Authorization HTTP header:
Authorization: "token 1234"
  • In case of a POST request, along with the request body:
key: 1234

Managing tokens

Generating a new token

You can generate a new token by running the following command:

curl -X POST --data name=token-name https://assets.ubuntu.com/v1/tokens?token={your-exisiting-token}

Removing a token

You can remove a token by running the following command:

curl -X DELETE https://assets.ubuntu.com/v1/tokens/{token-to-be-delete}?token={your-token}

Listing all the tokens

You can list all the tokens by running the following command:

curl https://assets.ubuntu.com/v1/tokens?token={your-token}

Managing assets

Uploading assets

You can upload assets with the cryptically named upload-assets tool. This can be installed with snap install upload-assets or sudo pip3 install upload-assets.

It's usually best to store your API key in your RC file (e.g. ~/.bashrc) by adding a line like export UPLOAD_ASSETS_API_TOKEN=xxxxxx. (You then probably need to source ~/.bashrc to load the environment variable).

You can then upload assets:

$ upload-asset --api-domain localhost:8017 MY-IMAGE.png
{'url': u'http://localhost:8017/v1/xxxxx-MY-IMAGE.png', 'image': True, 'created': u'Tue Sep 27 16:13:22 2016', 'file_path': u'xxxxx-MY-IMAGE.png', 'tags': u''}

You can also directly upload assets using the API:

echo "asset=$(base64 -w 0 MY-IMAGE.png)" | \
  curl --request POST --data @- --data "friendly-name=MY-IMAGE.png" "https://assets.ubuntu.com/v1/?token={your-api-token}"

In the example above, we used an option called friendly-name which is an option among others:

  • asset: (required) The base64 encoded asset
  • friendly-name: (optional) The name of the asset to be included in the asset's URL
  • url-path: (optional, default: the SHA1 of the asset) The path of the asset as it will be served over HTTP
  • optimize: (optional, default: false) Whether to optimize the image, only works for images of type PNG, JPEG and SVG, this option is ignored for other types of assets
  • tags: (optional, default: []) A comma separated list of tags to be associated with the asset

Deleting assets

Warning: Please read this before deleting anything

The assets server serves all assets with a cache-control header instructing all clients to cache the asset for a year. This is to get the best possible performance on our websites. You need to bear this in mind before deleting any assets. If the asset has been cached by any clients - from our own Content Cache, to other intermediary caches, to users' browsers - you may struggle to get the assets deleted from those caches.

For this reason it's best to find ways around needing to regularly delete assets.

This is also why the assets manager doesn't support deleting assets through the interface.

To delete an assets, simply use curl with the DELETE method:

curl --request DELETE "https://assets.ubuntu.com/v1/{asset-filename}?token={your-api-token}"

Listing assets

You can list all the assets by running the following command:

curl https://assets.ubuntu.com/v1/?token={your-api-token}

You can also filter the assets by:

  • tag: Filter the assets by a specific tag
  • q: A query string to filter the assets by filename, e.g. q=ubuntu will return all the assets with ubuntu in their filename
  • type: The type of the asset, e.g. type=png will return all the assets with the png extension

Managing redirects

Since assets are cached for a very long time, if you know you will want to update the version of an assets behind a specific URL, this should be achieved by setting up a (non-permanent) redirect to the assets.

E.g., Every day we upload the latest Server Guide to e.g. https://assets.ubuntu.com/v1/25868d7a-ubuntu-server-guide-2022-07-11.pdf, and then we change the https://assets.ubuntu.com/ubuntu-server-guide URL to redirect to this latest version.

Creating redirects

You can set up a new redirect with curl --data redirect_path={the-path-to-redirect} --data target_url={the-redirect-target} https://assets.ubuntu.com/v1/redirects?token={your-api-token}.

E.g. this would create the https://assets.ubuntu.com/ubuntu-server-guide redirect mentioned above:

$ curl --data redirect_path=ubuntu-server-guide --data target_url=https://assets.ubuntu.com/v1/25868d7a-ubuntu-server-guide-2022-07-11.pdf "https://assets.ubuntu.com/v1/redirects?token=xxxxxxxxxxx"
{
    "permanent": false,
    "message": "Redirect created",
    "target_url": "https://assets.ubuntu.com/v1/25868d7a-ubuntu-server-guide-2022-07-11.pdf",
    "redirect_path": "ubuntu-server-guide"
}

Updating redirects

Once a redirect already exists, you can use the PUT method to update it using curl --request PUT --data target_url={target-url} https://assets.ubuntu.com/v1/redirects/{redirect-path}?token={your-api-token}.

E.g. (following the above example):

$ curl --request PUT --data target_url=https://assets.ubuntu.com/v1/fe8d7514-ubuntu-server-guide-2022-07-13.pdf "https://assets.ubuntu.com/v1/redirects/ubuntu-server-guide?token=xxxxxxxxx"
{
    "target_url": "https://assets.ubuntu.com/v1/fe8d7514-ubuntu-server-guide-2022-07-13.pdf",
    "permanent": false,
    "redirect_path": "ubuntu-server-guide"
}

Deleting redirects

Deleting redirects is similarly simple, with curl --request DELETE https://assets.ubuntu.com/v1/redirects/{redirect-path}?token={your-api-token}, e.g. curl --request DELETE https://assets.ubuntu.com/v1/redirects/ubuntu-server-guide?token=xxxxxxxxx.

Security

As the server only uses a basic token for authentication, it is paramount that in a production setting, the API functions are only accessed over HTTPS, to keep the API token secret. For this reason, when DEBUG==false the server will force a redirect to HTTPS for all API calls.

Caching

The server is intended to be run in production behind a caching layer (e.g. squid cache). And as the server stores assets by default with a unique hash corresponding to the file's contents (e.g. a2f56da4-some-image.png), the cache expiration time should be as long as possible to maximise performance.

assets.ubuntu.com's People

Contributors

0atman avatar anthonydillon avatar b-m-f avatar barrymcgee avatar caleb-ellis avatar carkod avatar goulinkh avatar jkfran avatar jpmartinspt avatar meltysnow avatar mtruj013 avatar nottrobin avatar petesfrench avatar renovate-bot avatar samhotep avatar sowasred2012 avatar steverydz avatar willmoggridge avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

assets.ubuntu.com's Issues

Add CORS headers to redirects for fonts

We have a problem that if a site is pointing at a redirect for a font, it won't load the fonts properly because the original URL doesn't include the header Access-Control-Allow-Origin: *, meaning browsers will encounter a security block before they can follow the redirect.

So add Access-Control-Allow-Origin: * to redirects for fonts.

Add database and swift to CI setup

The tests are currently failing in our flask version because we don't have all the necessary services set up in circle CI. We investigate how to do this with circle CI.

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

  • Update all dependencies (Babel, Django, Jinja2, Wand, Werkzeug, itsdangerous, mongomock, more-itertools, netaddr, prettytable, pymongo, python-keystoneclient, python-magic, python-swiftclient, pytz, requests, scour, sh, simplejson, talisker, tornado)

  • Check this box to trigger a request for Renovate to run again on this repository

ImportError: No module named django.conf

Trying to set up asset server to test at home and getting:

$ scripts/upload-asset.py '/home/anthonydillon/Downloads/openstack-roadshow-globe.png' 
Traceback (most recent call last):
  File "scripts/upload-asset.py", line 12, in <module>
    from lib.file_helpers import create_asset
  File "/home/anthonydillon/Documents/Canonical/Git/assets-server/assets_server/lib/file_helpers.py", line 5, in <module>
    from django.conf import settings
ImportError: No module named django.conf

On Ubuntu 14.04 LTS.

Allow all origins for fonts

We need to allow people to link to fonts on assets.ubuntu.com. Therefore the assets server should send Access-Control-Allow-Origin: * along with any font types (especially woff)

Image optimisation

For binary images, run them through an optimiser on upload.

For SVGs, upload them as normal, but when they're requested by default they should be minified by stripping out pointless information, but there should be a query parameter for requesting the image unminified.

Given that we're now doing a fair few different things to process assets as they're being served, we should encapsulate this nicely by creating a nice way of defining asset processors at serve-time or upload-time.

Support for .woff2

.woff2 files should be recognised as "application/font-woff" mimetype.

We could do this through some intelligent parsing of the content, or if that doesn't work, by just adding the extra extension.

Fix assets security issues in staging

From elmo on IRC:

can someone please look at juju-stg-comms-assets-machine-{2,3,4,5,6}? they're missing security updates and Landscape can't fix it

Tests not running

./run test does not execute any tests, while being stated in the README.md as the command to execute them.

Support client-hints?

It may be helpful to add support for client hints. If we do this, I think it should be turned on with a flag in the URL, like client-hints=on or something.

Though this sounds like a promising option I'm not sure how useful it'll be in practice because infinitely scaling an image for every viewport out there means that we can't take advantage of caching, and ultimately I think caching may be more important than having the perfectly-sized image in almost every case.

@anthonydillon thoughts?

File size information in API

Can we add the file-size of the uploaded image as an API-accessible attribute, please.

The use-case for this would be download listings where a series of files and their stats are presented.

Add missing favicon.ico

Hi,

Can we add the missing favicon or a 301 permanent redirect to one?

$ curl -I https://assets.ubuntu.com/favicon.ico
HTTP/2 404
server: nginx/1.14.0 (Ubuntu)
date: Fri, 01 May 2020 21:59:17 GMT
content-type: application/json
allow: GET, HEAD, OPTIONS
x-cache: MISS from juju-prod45-ubuntu-assets-machine-10
x-cache-lookup: MISS from juju-prod45-ubuntu-assets-machine-10:80
via: 1.1 juju-prod45-ubuntu-assets-machine-10 (squid/3.3.8)

Thanks.

Switch over to Github actions

We can do this after the flask migration is done. Currently we are still using Circle CI, and for some reason we didn't switch over to Github actions. Worth looking into.

Is PNG conversion working?

https://assets.staging.ubuntu.com/v1/1163d53e-NewTux.svg -> works
https://assets.staging.ubuntu.com/v1/1163d53e-NewTux.svg.png -> 500!

How was this supposed to work anyway? Can you convert both ways? I think maybe you should be able to, but maybe we should change the API, for similar reasons as mentioned in #37?

Maybe it should be /my-asset.jpg?convert-to=png, and then also support conversion the other way, so that people can upload in either format. We would have to check that this API will work okay with front-end devs - do they rely on a file extension (.png) for some of their libraries?

Move the assets server to Kubernetes

For ubuntu.com:
Compressing resources with gzip or deflate can reduce the number of bytes sent over the network.

Enable compression for the following resources to reduce their transfer size by 13.6KiB (65% reduction).

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

File: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Configuration option compatibility.python should be a json object

Remove `django-error.log` from repository

This file's existence in the repo is annoying, because if keeps showing up as changed in local git repositories, and we have to be careful not to commit it.

It is only in the repository to overcome an annoying complication in the juju deployments. We should find another way around this, so we can remove this file from the repository and add it to .gitignore.

Tags searching

  • Store comma-separated tags as an array rather than a string in mongodb
  • Store unique tags in a new "tags" mongodb collection (for future development possibilities)
  • Work out how to migrate existing tags strings to be arrays
  • Add a method for searching by tag (/v1/?tag={tag} probably)

Prevent binary images from being expanded

Producing large images puts significant strain on the server (e.g.: all-the-things.jpg?w=90000 - except now that's bloody cached innit? To see what I mean change that number to something unique), so it's a potential concern if people start requesting a lot of uncached large image resizes. This is also therefore a potential attack vector.

@anthonydillon and I think we should just prevent the server producing images larger than their original upload size. I can't think of a case when you'd want this anyway, as you'll always be degrading quality, but increasing the download file size anyway. If you really need it larger you could expand it client-side (<img width="9000"...).

So, for binary images we should return a "400 Bad Request" with an informative error message if either the requested width or height is bigger than the original image. The error message should say what the maximum allowed width and height is.

Change the JSON data URL method (.json => /info)

Currently you can get information about an individual asset by accessing /my-image.jpg.json. This works, but is it the clearest way to do it? My concern is that it's not easy to tell whether you're expecting a JSON file or JSON about that file. What does happen if you upload a JSON file? Especially one which has multiple extensions (my-json.2014.json).

Maybe one of:

  • /data/my-image.jpg
  • /my-image.jpg/info
  • my-image.jpg?format=json

Would be clearer?

Add image dimensions to /info

This would allow the manager or any other frontend to find and display width and height information for an asset.

Improve error output from upload-healthtest.sh

If upload-healthtest.sh fails in during a release, it's incredibly difficult to debug. This actually goes for all our scripts, and run.py.

Try to improve the error messaging. Also simplify the script if you can. Maybe remove "upload-healthtest.sh" and run ".upload-asset.py" directly? Why do we have all these hidden scripts anyway?

Gzip and Brotli compression for assets

Assets should definitely be compressed with gzip and brotli. At the moment it appears that even Gzip it's not enabled at all for any assets.

This will be easily achieved by moving the application to Kubernetes, which we hope to do soon, but we should keep an issue here for it to remind ourselves too.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.