Code Monkey home page Code Monkey logo

async-disk-cache's Introduction

async-disk-cache Build status Build Status

An async disk cache. inspired by jgable/cache-swap

A sync sibling version is also available: stefanpenner/sync-disk-cache

By default, this will usge TMPDIR/<username>/ for storage, but this can be changed by setting the $TMPDIR environment variable.

Example

var Cache = require('async-disk-cache');
var cache = new Cache('my-cache');
// 'my-cache' also serves as the global key for the cache.
// if you have multiple programs with this same `cache-key` they will share the
// same backing store. This by design.

// checking
cache.has('foo').then(function(wasFooFound) {

});

// retrieving (cache hit)
cache.get('foo').then(function(cacheEntry) {
  cacheEntry === {
    isCached: true,
    key: 'foo',
    value: 'content of foo'
  }
});

// retrieving (cache miss)
cache.get('foo').then(function(cacheEntry) {
  cacheEntry === {
    isCached: false,
    key: 'foo',
    value: undefined
  }
});

// setting
cache.set('foo', 'content of foo').then(function() {
  // foo was set
});

// clearing one entry from the cache
cache.remove('foo').then(function() {
  // foo was removed
})

// clearing the whole cache
cache.clear().then(function() {
  // cache was cleared
})

Enable compression:

var Cache = require('async-disk-cache');
var cache = new Cache('my-cache', {
  compression: 'gzip' | 'deflate' | 'deflateRaw', // basically just what nodes zlib's ships with
  supportBuffer: 'true' | 'false' // add support for file caching (default `false`)
})

HELP!...my TMP dir is growing unbounded!

description

In general most OS distributions come with cron like tasks, which purge unused files in $TMPDIR. For example, ubuntu typically uses tmpreaper and macOS uses various tasks in /etc/periodic/*.

options

If your OS distribution does not provide such a cleanup mechanism:

a) We stronglly recommend utilizing one, as other sync-disk-cache is not alone in rely on this behavior b) If that is not possible, we recommend changing your $TMPDIR to something project specific and manually purging it.

License

Licensed under the MIT License, Copyright 2015 Stefan Penner

async-disk-cache's People

Contributors

chriseppstein avatar greenkeeperio-bot avatar keithws avatar mfogel avatar rwjblue avatar stefanpenner avatar williscool avatar yongk802 avatar zebraflesh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

async-disk-cache's Issues

Add environment variable to override usage of `os.tmpDir()`.

This would allow us to utilize a known path as the cache location, and then we can utilize Travis caching to take advantage of warm boots in CI also.

I would like https://github.com/stefanpenner/async-disk-cache/blob/master/index.js#L81 to become:

this.tmpDir = options.location || process.env['ASYNC_DISK_CACHE_ROOT'] || tmpDir;

Then we could use a path that is in the CI services (both Travis and Circle support these options) cached directory listing, and get much faster CI builds...

Infinite loop on windows when key includes invalid characters for filesystem

root cause of kimroen/ember-cli-coffeescript#112
broccoli-coffee uses the characters ", : in the key, which when ultimately called in writeP
infinitely recurses as reason.code is ENOENT

https://github.com/stefanpenner/async-disk-cache/blob/master/index.js#L173

you should be able to reproduce (on windows) with

AsyncDiskCache = require('async-disk-cache')
cache = new AsyncDiskCache
cache.set('foo/:/1', 'bar')

hmm, not sure this is really an issue for this project.
it seems the infinite loop is within mkdirp, there is a pending PR from a couple years ago on mkdirp that fixes this https://github.com/substack/node-mkdirp/pull/74/commits

No cache created when used on Github actions

We are benchmarking our ember.js app build time and to monitor this, we have a nightly github actions workflow that runs ember build twice and logs the build time.

I expected the second ember build to be faster than the first due to the build cache being present, but this is not the case. Both builds take the same time…

After some logging, it looks like cache folder is not created, and if I created one manually and use its path with TMPDIR=/my/tmp/path ember build, then it won't be populated.

Is this a know behaviour with Github actions?

instrumentation cache metrics

Track the metrics (eg. cache hits) and that will show in the instrumentation JSON when instrumentation (BROCCOLI_VIZ/EMBER_CLI_INSTRUMENTATION) is enabled.

improve docker build with if-you-need-to-delete-this-open-an-issue-async-disk-cache

I have a docker image that takes 13 minutes to run ember build (fairly large ember project). Running on the host machine will also take 13 minutes initially, but drops down to a more manageable 1 minute build time after initial build (using the cache stored in /tmp/{user}/if-you-need-to-delete-this-open-an-issue-async-disk-cache i pressume). I tried copying in the cache into my docker build prior to running the ember build, but it doesn't seem like it uses the cache (continues to take ~13 minutes). Any ideas on whether the cache is tied to something built in my local repository or improving my docker build time in general?

v1.1.0 is missing ./lib/metric.js

v1.1.0 is the latest version I can install from npm

Error: Cannot find module './lib/metric'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/jbryson3/code/arbiter2/node_modules/async-disk-cache/index.js:18:14)

Metrics must support concurrent usage.

Issue for: #49

fs.writeFile(x, err => /*...*/); // <— calls start (but no stop yet)
fs.writeFile(y, err => /*...*/); // <— call start (but no stop yet)

// some time in the future
// stop is called
// stop is called again, but \w sadness https://github.com/stefanpenner/async-disk-cache/pull/49 works around

I believe this can be solved by a counter.

  • increment metric.start()
  • decrement metric.stop()

When the counter === 0, truly stop.

Using up inode limit

We are building a medium-sized Ember application in a CI environment running Jenkins on an AWS EC2 instance. We are finding that /tmp/if-you-need-to-delete-this-open-an-issue-async-disk-cache never gets cleaned out and continues to grow until we run out of inodes. Clearing it today brought our inode usage on the machine down from 95% to 30%. This is causing all our other builds to fail as Jenkins is no longer able to create new files.

Using all the inodes

This is a follow up of #32 which was closed without a real solution.

We had the same issue with a jenkins job building an ember application.
The cache folder was not in /tmp, because each job should only write into its workspace.

So broccoli-persistent-filter and async-disk-cache keep putting files without ever cleaning them.
It's not the task of the OS to clean up files left by a software running on it.

Can we expect a solution which is not letting the OS clean the garbage left?

Race condition when sharing cache

If the persistent cache is being accessed by more than one process, it's possible to see partially-written records in the cache.

Instead we could write each new file to a globally unique temporary name and then atomically rename it to the real shared name.

Same issue on the sync sibling of this library: stefanpenner/sync-disk-cache#18

Error: ENAMETOOLONG: name too long

Sometimes, async-disk-cache generates filenames that are longer than 255 characters. This causes HFS+ (on macOS) to throw the ENAMETOOLONG error.

After reviewing the code, it looks the the file name is just the key in Base64, so obviously long keys will produce long filenames.

Also, Base64 seems problematic since it may include "/" characters which will be parsed as a directory separator.

This problem is typically solved by producing a hash of the key and storing it in hex, or less common, Base62.

Store Buffer

Hi!

How do i should properly store Buffer?
Can't find proper way to store Buffer : /

heimdall.hasMonitor is not a function

On trying to build I end up with TypeError: heimdall.hasMonitor is not a function

Using [email protected] it appears as though it is depending on heimdall.hasMonitor being a function, and appears as though that version of heimdall ([email protected]) doesn't expose heimdall.hasMonitor.

Here's the dep tree:

├─┬ [email protected]
│ ├─┬ [email protected]
│ │ └─┬ [email protected]
│ │   ├─┬ [email protected]
│ │   │ └── [email protected]

package.json calls for heimdalljs ^0.2.3 so I'm not sure how this even ever worked?

Error in example (content instead of value)

There's an error in the example code given in README.md ... the property holding the cache entry's content is named value, not content.

Thus the example should look like this:

cache.get('foo').then(function(cacheEntry) {
  cacheEntry === {
    isCached: true,
    path: 'foo',
    value: 'content of foo'
  }
});

Fails when invalid character used.

Maybe it's not the right place to report this bug, but actually I do have to delete the "if-you-need-to-delete-this-open-an-issue-async-disk-cache" directory.

Disclaimer, I don't know what the author have done with their code, but as far as I know, the usage of invalid character in Windows system always fails.

something related to async disk cache

Use a consistent base path to allow easier clearing of cache folders.

To allow easier clearing of the cache, I would like to store all caches for async-disk-cache in a subdirectory of this.tmpDir instead of its root. This would not add significant cost to the build times, but it would allow us to know what directories in tmpDir are from async-disk-cache.

Specifically, I would suggest the following change here:

this.root = path.join(this.tmpDir, 'async-disk-cache', this.key);

Windows infinite loop / performance

Hello...

I am experiencing a slow build/run performance issue with Ember-cli on Windows 10. I attempted running the command shell as administrator, disabled indexing on temp folders, and attempted to use the ember-cli-windows module to change system settings.

I used Process Monitor to inspect what was happening further. It appears that the async-disk-cache process is trying to write out an invalid filename and is caught in an infinite loop. I can see in Process Monitor, these events happening in a cycle:

Time Process Name Pid Operation Path Result Detail
11:53:38.0171771 AM node.exe 5444 CreateFile C:\Users\nivenhuh\AppData\Local\Temp\if-you-need-to-delete-this-open-an-issue-async-disk-cache\a31bd73b8eda8ffd76238ecb16985611d7a9d184{"bare":true,"destDir":"myEmberApp","persist":true,"srcDir":" NAME INVALID Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Open Reparse Point, Attributes: N, ShareMode: Read, Write, AllocationSize: 0
11:53:38.0173547 AM node.exe 5444 CreateFile C:\Users\nivenhuh\AppData\Local\Temp\if-you-need-to-delete-this-open-an-issue-async-disk-cache\a31bd73b8eda8ffd76238ecb16985611d7a9d184 NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Open Reparse Point, Attributes: N, ShareMode: Read, Write, AllocationSize: 0
11:53:38.0175284 AM node.exe 5444 CreateFile C:\Users\nivenhuh\AppData\Local\Temp\if-you-need-to-delete-this-open-an-issue-async-disk-cache\a31bd73b8eda8ffd76238ecb16985611d7a9d184 SUCCESS Desired Access: Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
11:53:38.0175493 AM node.exe 5444 QueryAllInformationFile C:\Users\nivenhuh\AppData\Local\Temp\if-you-need-to-delete-this-open-an-issue-async-disk-cache\a31bd73b8eda8ffd76238ecb16985611d7a9d184 BUFFER OVERFLOW CreationTime: 2/20/2017 11:46:55 AM, LastAccessTime: 2/20/2017 11:47:34 AM, LastWriteTime: 2/20/2017 11:47:34 AM, ChangeTime: 2/20/2017 11:47:34 AM, FileAttributes: D, AllocationSize: 786,432, EndOfFile: 786,432, NumberOfLinks: 1, DeletePending: False, Directory: True, IndexNumber: 0x400000005c61d, EaSize: 0, Access: Read Attributes, Synchronize, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Word
11:53:38.0175604 AM node.exe 5444 QueryInformationVolume C:\Users\nivenhuh\AppData\Local\Temp\if-you-need-to-delete-this-open-an-issue-async-disk-cache\a31bd73b8eda8ffd76238ecb16985611d7a9d184 BUFFER OVERFLOW VolumeCreationTime: 7/6/2016 8:50:12 AM, VolumeSerialNumber: 2C76-E880, SupportsObjects: True, VolumeLabel: WINă
11:53:38.0175691 AM node.exe 5444 CloseFile C:\Users\nivenhuh\AppData\Local\Temp\if-you-need-to-delete-this-open-an-issue-async-disk-cache\a31bd73b8eda8ffd76238ecb16985611d7a9d184 SUCCESS

One thing of note is the json string being written out as a file name. I'm not sure if this is supported? (The NAME INVALID) error from Process Monitor.

>ember --version
ember-cli: 2.11.1
node: 6.9.5
os: win32 x64

if-you-need-to-delete-this-open-an-issue-async-disk-cache

Just following orders :)

I see this error pop up often when starting a large Ember app.

Error: ENFILE: file table overflow, open '/var/folders/ds/rbrsq85d0y52f1j1ybxws791q71g36/T/mitch/if-you-need-to-delete-this-open-an-issue-async-disk-cache/5711b76f73750a8c95630eb9be2bcadaea1db8db/031bdfabd03a1f186739db24041795c423f2a2e1

Temporary files are created with access rights that are too permissive

It seems like temp directories are created with mode 0775 (group-writable) and files are created with mode 0666 (group-writable and world-writable). This was brought to our attention during a security audit.

Is there any reason this cannot be made more restrictive? On the face of it, it seems like a security risk to allow other users and processes to make changes to your temporary files.
Temporary files should really just be writeable by the current user (mode 0700 for directories and mode 0600 for files).

Or the code could just to respect the user's umask - that would probably be best.

Breaking change done in the ongoing version update

I am creating this issue because I noticed that a breaking change was made in a minor version release of the project

Recently async-disk-cache project updated username library via commit 9113a49. After this commit and v1.2.1 release, all projects that are using ember-cli-htmlbars, ember-cli-babel, etc and using node version older than 4.0.0 are failing when doing "node ember build".

When releasing new major version 2.0.0, "sindresorhus/username" made a breaking change via commit. This is completely acceptable on part of "sindresorhus/username" because on making that change, they made a major version change from 1.0.1 to 2.0.0. So projects that are depending on "sindresorhus/username" were not automatically affected because of this breaking change and thus no cascading effect on other projects.

But if stefanpenner/async-disk-cache is bumping up sindresorhus/username version from 1.0.1 to 2.3.0 then stefanpenner/async-disk-cache version should have also been updated from v1.2.0 to v2.0.0 instead of v1.2.1 to avoid any cascading effect. That would have not created build-breaking issues for libraries using stefanpenner/async-disk-cache and using node versions older than 4.0.0.

It will be really helpful if someone can look into this issue.

-- Thanks

Remove extra `mkdirp` calls.

We are currently calling mkdirp for every cache set operation to ensure that the containing directory is created before writing the file (here). Unfortunately, this results in a ton of extra file system calls (mkdirp recursively calls fs.exists for each segment in the provided path even if the path exists).

We should instead, attempt to write to the path and if an error (generally ENOENT) occurs then call mkdirp.

`EACCES: permission denied, mkdir '/tmp/root/if-you-need-to-delete-this-open-an-issue-async-disk-cache/***'`

Trying to upgrade a ghost website, upgrade fails at the line mentioned in this issue title.
Ghost is installed via yunohost and the ghost_ynh package.
From my understanding, this package is creating a ghost__X user (where X is a number, so that multiple websites can be installed on the same server). I would expect therefore the async-disk-cache folder to be created in /tmp/ghost__X/ and not in /tmp/root/.
Could there be something in async-disk-cache that somehow misrecognize the user doing the action?

whoami problem on windows with yarn

A couple of weeks ago the code started to use 'whoami' instead of the username node module: 019f242

Since then I can no longer use it using yarn on my Windows 7 machine in the context of an ember build.
Version: async-disk-cache@^1.0.0: version "1.3.1"

yarn run serve
yarn run v0.23.4
$ ember serve --proxy http://127.0.0.1:8080
'whoami' is not recognized as an internal or external command,
operable program or batch file.
File: C:\Windows\system32\cmd.exe
Command failed: whoami
'whoami' is not recognized as an internal or external command,
operable program or batch file.

Error: Command failed: whoami
'whoami' is not recognized as an internal or external command,
operable program or batch file.

    at checkExecSyncError (child_process.js:481:13)
    at Object.execSync (child_process.js:521:13)
    at Object.<anonymous> (C:\....\node_modules\async-disk-cache\index.js:25:41)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

if-you-need-to-delete-this-open-an-issue-async-disk-cache size keeps growing

We are using Jenkins to build our Ember application.
A temporary GUID name file keeps growing its size and eat up all the inodes in our server in /tmp/if-you-need-to-delete-this-open-an-issue-async-disk-cache folder.

Can you please help to resolve this issue? Thanks!

Additionally, what if I delete this folder? Do any bad things gonna happen during the CI?
Or do we have an elegant solution for it?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.