Code Monkey home page Code Monkey logo

browscap-site's Introduction

Browser Capabilities Project

Continuous Integration codecov

This tool is used to build and maintain browscap files.

Installation

$ git clone git://github.com/browscap/browscap.git
$ cd browscap
$ curl -s https://getcomposer.org/installer | php
$ php composer.phar install

What's changed in version 6048

  • #2535 Added recent new Apple platforms (Mac OS, iOS and iPadOS)

What's changed in version 6028

BC breaks listed

  • Interface changed for class \Browscap\Data\Factory\UserAgentFactory

What's changed in version 6027

BC breaks listed

  • Strict type hints have been added throughout. This may break some type assumptions made in earlier versions.
  • In many classes Setters and Getters have been removed, the parameters have been moved to the class constructor
  • Some classes are now final - use composition instead of inheritance

What's changed in version 6025

BC breaks listed

  • The grep command and the diff command were removed

Changes

  • The tests for integration testing the source files are split from the other tests
  • Tests on travis use the build pipeline now

Directory Structure

  • bin - Contains executable files
  • build - Contains various builds
  • resources - Files needed to build the various files, also used to validate the capabilities
  • src - The code of this project lives here
  • tests - The testing code of this project lives here

the CLI commands

There is actually only one cli command available.

build

This command is used to build a set of defined browscap files.

bin/browscap build [version]

options

  • version (required) the name of the version that should be built
  • output (optional) the directory where the files should be created
  • resources (optional) the directory where the sources for the build are located
  • coverage (optional) if this option is set, during the build information is added which can be used to generate a coverage report
  • no-zip (optional) if this option is set, no zip file is generated during the build

For further documentation on the build command, see here.

CLI Examples

You can export a new set of browscap files:

$ bin/browscap build 5020-test
Resource folder: <your source dir>
Build folder: <your target dir>
Generating full_asp_browscap.ini [ASP/FULL]
Generating full_php_browscap.ini [PHP/FULL]
Generating browscap.ini [ASP]
Generating php_browscap.ini [PHP]
...
All done.
$

Now you if you look at browscap/browscap.ini you will see a new INI file has been generated.

Usage Examples

How to build a standard set of browscap files

This example assumes that you want to build all *php_browscap.ini files.

$logger = new \Monolog\Logger('browscap'); // or maybe any other PSR-3 compatible Logger

$format = \Browscap\Formatter\FormatterInterface::TYPE_PHP; // you may choose the output format you want, the format must be already supported

$resourceFolder = 'resources/'; // please point to the resources directory inside the project
$buildFolder = ''; // choose the directory where the generated file should be written to

// If you are using one of the predefined WriterFactories, you may not choose the file names
$writerCollection = (new \Browscap\Writer\Factory\PhpWriterFactory())->createCollection($logger, $buildFolder);

$dataCollectionFactory = new \Browscap\Data\Factory\DataCollectionFactory($logger);

$buildGenerator = new BuildGenerator(
    $resourceFolder,
    $buildFolder,
    $logger,
    $writerCollection,
    $dataCollectionFactory
);

$version       = '';    // what you want to be written into the generated file
$createZipFile = false; // It is not possible yet to create a zipped version of a custom named browscap file

$buildGenerator->run($version, $createZipFile);

How to build a custom set of browscap files

If you want to build a custom set of browscap files, you may not use the predefined WriterFactories.

$logger = new \Monolog\Logger('browscap'); // or maybe any other PSR-3 compatible Logger

$format = \Browscap\Formatter\FormatterInterface::TYPE_PHP; // you may choose the output format you want, the format must be already supported

$resourceFolder = 'resources/'; // please point to the resources directory inside the project
$buildFolder = ''; // choose the directory where the generated file should be written to

$propertyHolder = new \Browscap\Data\PropertyHolder();

// build a standard version browscap.json file
$jsonFormatter = new \Browscap\Formatter\JsonFormatter($propertyHolder);
$jsonFilter    = new \Browscap\Filter\StandardFilter($propertyHolder);

$jsonWriter = new \Browscap\Writer\JsonWriter('relative path or name of the target file', $logger);
$jsonWriter->setFormatter($jsonFormatter);
$jsonWriter->setFilter($jsonFilter);

// build a lite version browscap.xml file
$xmlFormatter = new \Browscap\Formatter\XmlFormatter($propertyHolder);
$xmlFilter    = new \Browscap\Filter\LiteFilter($propertyHolder);

$xmlWriter = new \Browscap\Writer\XmlWriter('relative path or name of the target file', $logger);
$xmlWriter->setFormatter($xmlFormatter);
$xmlWriter->setFilter($xmlFilter);

$writerCollection = new \Browscap\Writer\WriterCollection();
$writerCollection->addWriter($jsonWriter);
$writerCollection->addWriter($xmlWriter);

$dataCollectionFactory = new \Browscap\Data\Factory\DataCollectionFactory($logger);

$buildGenerator = new BuildGenerator(
    $resourceFolder,
    $buildFolder,
    $logger,
    $writerCollection,
    $dataCollectionFactory
);

$version       = '';    // what you want to be written into the generated file
$createZipFile = false; // It is not possible yet to create a zipped version of a custom named browscap file

$buildGenerator->run($version, $createZipFile);

How to build a custom browscap.ini

If you want to build a custom browscap file you may choose the file name and the fields which are included.

Note: It is not possible to build a custom browscap.ini file with the CLI command.

$logger = new \Monolog\Logger('browscap'); // or maybe any other PSR-3 compatible Logger
// If using Monolog, you need specify a log handler, e.g. for STDOUT: $logger->pushHandler(new \Monolog\Handler\ErrorLogHandler());

$format = \Browscap\Formatter\FormatterInterface::TYPE_PHP; // you may choose the output format you want, the format must be already supported
$file   = null; // you may set a custom file name here
$fields = []; // choose the fields you want inside of your browscap file

$resourceFolder = 'resources/'; // please point to the resources directory inside the project
$buildFolder = ''; // choose the directory where the generated file should be written to

$writerCollection = (new \Browscap\Writer\Factory\CustomWriterFactory())->createCollection($logger, $buildFolder, $file, $fields, $format);

$dataCollectionFactory = new \Browscap\Data\Factory\DataCollectionFactory($logger);

$buildGenerator = new BuildGenerator(
    $resourceFolder,
    $buildFolder,
    $logger,
    $writerCollection,
    $dataCollectionFactory
);

$version       = ''; // version you want to be written into the generated file
$dateTime      = new \DateTimeImmutable(); // date you want to be written into the generated file
$createZipFile = false; // It is not possible yet to create a zipped version of a custom named browscap file

$buildGenerator->run($version, $dateTime, $createZipFile);

Issues and feature requests

Please report your issues and ask for new features on the GitHub Issue Tracker at https://github.com/browscap/browscap/issues

Contributing

For instructions on how to contribute see the CONTRIBUTE.md file.

License

See the LICENSE file.

browscap-site's People

Contributors

asgrim avatar daawesomep avatar dependabot-preview[bot] avatar dependabot[bot] avatar github-actions[bot] avatar mimmi20 avatar scrutinizer-auto-fixer avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

browscap-site's Issues

Automatically check for updates when browscap/browscap is released

At the moment, Dependabot checks once a day, which is fine for normal dep updates. However, we want browscap/browscap releases to almost immediately trigger an update check here, so we can consume the new release.

It seems this isn't possible at the moment: dependabot/dependabot-core#3080 - if there was an API call available, @frankdejonge suggested:

If that is available on the GitHub API then you can create an action for it by using that base action which is just a JS hook

The URL it would call is https://github.com/browscap/browscap-site/network/updates?update_config_id=<some-id> - however, the POST payload seems to be some kind of number-used once called authenticity_token, so my guess is this isn't easily hackable (I mean, it probably is, but it'll likely be brittle).

A workaround was suggested on Twitter by @bendavies :

an awful workaround:
https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates
"When you add or update the dependabot.yml file, this triggers an immediate check for version updates."

so on tagging package A, trigger a workflow that updates repo B dependabot.yml in some way.

Whilst it may help, it seems a bit of a hack.

The alternatives at the moment:

  • Wait up to 24 hours for Dependabot to pick up the release and send the PR
  • Manually click the "Check for updates" button

It's worth noting that we'd have to introduce some kind of wait / polling of the Packagist API, since the release does NOT become immediately available (either when Dependabot checks, or occasionally I've seen Heroku not pick up the new release even!), so there will always be some lag whilst Packagist caches update, which we need to account for in any automated processes.

Feature request: deflate support

Originally reported by @AohRveTPV in browscap/browscap#320:

Hello, I have been trying to improve downloads of Browscap data in the Browscap Drupal module. It is already possible to use gzip compression with the download links, which reduces download size by a factor of 10-20x. However, gzip support is only available in PHP 5.4.0, and Drupal supports earlier versions of PHP.

If the browscap.org had support for deflate, all Drupal-supported PHP versions would support compression, and the Browscap Drupal module could exclusively use it. Since there are ~11000 sites using the Browscap module, a rough calculation indicates this would potentially save 65GB of traffic to http://browscap.org each time a new dataset is released.

I am not totally certain this would work, but wanted to put forth the feature request to see if it is something the Browscap project might be willing to add. http://browscap.org seems to be using Apache, and there is mod_deflate for Apache.

In any case, thank you for providing all the Browscap data.

Only put file downloads on cloudflare DDOS protection?

Based on some discussion from this ticket (browscap/browscap-php#228) it sounds like Cloudflare has DDOS protection on all of browscap.org in order to prevent abusers from eating too much bandwidth, but apparently this is happening for all requests to browscap.org, even the version endpoints.

Would it be possible to move the actual file download to a different sub-domain on browscap.org, and only enable the DDOS protection on that sub-domain, so that the version endpoints will still work for these users?

The download link would probably have to be changed to redirect to this sub-domain.

Submission of New User Agents

Reported initially by @dczaretsky in browscap/browscap#827

Would be great if there was an easy form on browscap.org to submit and fill in the info for new user agents. Maybe others can then help more to contribute and test. Seems the current method of posting to Github is a bit more cumbersome.

Just a thought.

UA Lookup not working

Whenever i try to perform a UA Lookup, it just returns

"Whoops, looks like something went wrong."

Possibility of a mirror?

We've discussed this briefly before (browscap/browscap#511 (comment)) but maybe we could figure out a good way to make this work?

Could the download requests still go to browscap.org for stat collection and abuse mitigation, but the actual file request would be distributed between browscap.org and a mirror we'd provide?

Abusers could still potentially just cache the final redirect, whether it's browscap.org or our mirror, but maybe it would reduce your bandwidth usage to have the legit users split between two file locations.

I'm not sure what the specifics would be yet, just wanted to bring it up for further discussion since bandwidth seems to be a problem lately.

add nofollow or noindex label to download links

If you search with Google for Mozilla/5.0 (compatible; fr-crawler/1.1) which is part of browscap/browscap#267 the link browscap.org/stream?q=BrowsCapINI is part of the search result.

I think, our ini files dont need to be indexed by Google. A rel="nofollow" should be added to each download link, or it should be added with a meta tag.

Upgrade to Composer 2

  • Check that Heroku supports Composer 2 first
  • Check we're compatible with Composer 2 (unsure if any BC breaks in hooks / API etc. that we use)

Use browscap-php 3 to perform background update of cache & UA lookup

From https://github.com/browscap/browscap/wiki/Public-release-procedure

It is then recommended to re-initialise the UA lookup tool. First, cd .cache && rm * then run any search - it will take a short while to download the browscap.ini and re-cache the file.

We are actually going against our own recommendations to cache the browscap.ini for the UA lookup tool. Add a tool that maybe runs after a composer install|update that removes the current cache and rebuilds it, so that the first time you run the UA lookup tool it does not hang for ages.

Use ocramius/package-versions and use "proper" versions for build numbers

May be able to use ocramius/package-versions to determine what the package version of browscap/browscap requested is, and generate the build number properly. Would mean using a proper version number though, I'm thinking perhaps we form it from the current one i.e. next is 6029 so would be come 6.0.29. That way we can track BC breaks in browscap/browscap by bumping major version, and each patch version can simply be the new build release (means we can also drop the stupid BUILD_NUMBER textfile!)

Then in composer.json here (for the site) can just do "browscap/browscap": "6.0.29" and that would translate to 6029. Note the "minor" version we'd drop, it doesn't count to the middle version, so example translations:

  • 6.0.29 => 6029
  • 7.0.0 => 7000
  • 7.3.12 => 7012
  • 7.4.104 => 7104

Otherwise we limit ourselves to 99 builds per minor, this way we have 999 builds at least.

Or alternatively, we make the build number get really big next version and use 3 digits for each major/minor/patch version:

  • 6.0.29 => 6000029 (i.e. 006.000.029)
  • 7.0.0 => 7000000
  • 7.3.12 => 7003012
  • 7.4.104 => 7004104

The latter might be more flexible actually.

Finally, the other approach is to drop the build number entirely, and just use the version as major/minor/patch as-is, though this might break lots of stuff that depend on the "build number" concept...

Issue API keys to download the INI files

Although so far the INI files are readily available, this is problematic and we get a huge amount of abuse.

My idea was this: in order to download the files, you must generate an API key. Generating the API key shouldn't need an account or anything, but we could probably use a Google reCaptcha or something.

The API keys should expire (maybe in 1 year or something), and probably makes sense to be a JWT as the key, then we don't even need to store anything.

Basically anything without an API key would now be blocked.

Add rate limiting / banning system

At the moment, there is no rate limiting on the new site. We need to build a system whereby we measure the downloads from any IP - if it's more than a certain amount, ban the IP.

Or something better maybe, I haven't thought too much about it :)

The old site that Moose made simply uses a .htaccess to ban IPs, so yeah.

Automate file release process?

I don't know what this process currently entails, so it might be naive of me to assume that it's something that could be automated, but I wanted to discuss the possibility.

Would it be possible to be able to release a new version of the browscap file by updating composer in this project and having travis/circleci build and deploy to Digital Ocean (or whatever it is you're hosting on)?

Might allow us to release on a more regular schedule and do it without your intervention (@asgrim).

Thoughts?

Import `manage-bans.sh` as usable CLI tool

This is the script I used on Linode to manage the bans and unban people. Need to push this so it's usable in Heroku (note: mysql creds is the blocker here - but we have environment variables we can butcher for the mysql creds)

#!/bin/bash

MYSQL="mysql -u??? -p??? browscap "
USAGE="Usage : $0"

if [ $# -lt 1 ]
then
  echo "$USAGE command [options]"
  echo ""
  echo "  list x.x.x.x       Lists bans for an IP"
  echo "  downloads x.x.x.x  Lists downloads for an IP"
  echo "  unban 123          Converts a perm ban to a temp ban"
  echo "  unban-ip x.x.x.x   Converts all perm bans to temp bans"
  exit 1
fi

case "$1" in
  'list' )
    if [ "$2" = "" ]
    then
      echo "$USAGE list ip"
      exit 1
    fi
    $MYSQL -e "SELECT * FROM banLog WHERE ipAddress='$2'"
  ;;
  'unban' )
    if [ "$2" = "" ]
    then
      echo "$USAGE unban banId"
      exit 1
    fi
    $MYSQL -e "UPDATE banLog SET isPermanent = 0 WHERE id=$2"
  ;;
  'unban-ip' )
    if [ "$2" = "" ]
    then
      echo "$USAGE unban-ip ip"
      exit 1
    fi
    $MYSQL -e "UPDATE banLog SET isPermanent = 0 WHERE ipAddress='$2'"
  ;;
  'downloads' )
    if [ "$2" = "" ]
    then
      echo "$USAGE downloads ip"
      exit 1
    fi
    $MYSQL -e "SELECT * FROM downloadLog WHERE ipAddress='$2'"
  ;;
esac

Integrate rate limiting into CloudFlare IP firewall

I did try to ask CloudFlare to help me estimate the real cost of enabling the rate limiting feature (as, if the 100,000,000+ requests/month we get were all genuine - they're not - it'd cost around $500/mo).

Therefore, had an idea. When a rate limit abuse happens, instead of blocking it at the application level, block it in CloudFlare so the traffic never reaches us. There does seem to be a limit on the number of IPs that can be blocked, so maybe we can "rotate" them as such.

Need to find out what the limit is and what we can do about it.

IP access blocked

We first noticed the block about 20 hours ago.

We have called the service the same way for 5+ years and this is the first time we've been blocked.

I would be surprised if we tried to download the csv file more than 30 times in any 1 hour period... but I suppose it could be possible if something went wrong which may have led to retries (by automated processes). Otherwise, it should never really go over 10 max.

I would appreciate it if you can please check. All calls are made from IP - 54.208.16.197. Many thanks!

$ curl -I "https://browscap.org/stream?q=BrowsCapCSV"
HTTP/2 403 
date: Fri, 08 Mar 2024 07:59:42 GMT

Delete older download logs

DBs in PaaS world are a bit less flexible on account of we pay for what we use; therefore need to limit the amount of download logs stored.

We have 1GB database, currently using "60%" (ClearDB on Heroku is not exactly clear about how much data that is exactly...).

Maybe 6mo or 12mo is a reasonable cutoff?

Mobile usability issues

Apparently there were some Mobile Usability issues detected by Google Search Console:

  • Text too small to read
  • Viewport not set
  • Clickable elements too close together

Add padding below footer

On a smaller screen, the dotted line meets the bottom of the screen. Add 10px padding or something to the bottom to space it a little.

Generate MD5 / SHA1 checksums for browscap files

I wish to automate the updating of the browscap.ini files on our production servers, but I can't determine a way to programmatically verify the integrity of the downloaded files. Can you please also generate checksums that are hosted alongside the browscaps with predictable URLs so that we can automate verification of file integrity?

Originally reported at browscap/browscap#107 by @c0ldfusi0nz

Automatic updates of browscap

We should automate this: https://github.com/browscap/browscap/wiki/Public-release-procedure

The process could probably be a script or something I can run periodically. The process is basically:

browscap/browscap

  • determine github milestone ID (e.g. 1.0.0 = 11)
  • make a new milestone for 1.0.1
  • move any remaining 1.0.0 tickets to 1.0.1
  • run changelog_generator.php -c ~/path_to_config.php -u browscap -r browscap -m 56 for notes
  • close milestone 1.0.0
  • git pull
  • git tag -s 1.0.0
  • message = changelog
  • git push origin 1.0.0
  • create github release for version

browscap/browscap-site

  • docker-compose run php-server composer require browscap/browscap 6.0.33
  • git commit -m "Update site to serve browscap 6.0.33"
  • git push

Use a framework?

Silex? Symfony? Something else?

The site is pretty ugly (code-wise) at the moment and could do with some conventional structure.

Stats page

Create a page with download statistics.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.