Code Monkey home page Code Monkey logo

skott's Introduction

skott

version npm Maintenance License: MIT

skott is a minimalist developer tool that can be used to efficiently generate directed graphs from your JavaScript/TypeScript/Node.js project. It can automatically collect metadata such as file size, third-party or builtin dependencies, detect circular dependencies, finding unused npm dependencies, help you building tools relying on graph data structures thanks to the exposed primitives. Many display modes exists (such as embedded interactive web application shown just below) but also other from the CLI (file-tree, graph). Static files (.svg, .png, .md, .json) can also be generated.

skott

skott exposes directed graphs primitives so that it can be used to implement tools on top of graph structures e.g. affected/incremental patterns as it exposes a way to know precisely and deeply dependencies for each graph node. More generally speaking, anything that can be implemented using graphs can be implemented using skott.

✅ Works for modern JavaScript/TypeScript projects (TSX/JSX, ECMAScript and CommonJS modules all supported). TypeScript's path aliases are also supported.

✅ Deeply detects circular dependencies in an efficient way, with the ability to provide a max depth for the search

✅ Many builtin visualization modes including a web application or terminal-based outputs such as file-tree or graph views.

✅ Builtin watch mode updating the graph when file changes are detected. It works with all display modes (webapp and all CLIs visualization modes). Support all options of file ignoring/filtering from skott.

✅ File tree traversal supports ignore pattern and Git ignored files (via .gitignore files)

✅ Runs incrementally (experimental), meaning that skott will only analyze files that were modified/added since the last run, offering performance boosts.

✅ Works with any custom dependency resolver (useful for specific monorepos integration where module identifiers need to be mapped to a specific workspace package)

✅ Detect unused npm third-party dependencies. Note that all unused devDependencies are not guaranteed to be detected as depcheck only provides analysis for set of supported libraries (eslint, karma, mocha, etc).

✅ Deeply collect all dependencies of the project graph, including third-party and builtin.

✅ Deep parent and child dependencies traversals using DFS and BFS algorithms.

✅ Metadata collection per traversed node (file size, dependencies)

✅ Generate static files including raw JSON, mermaid-js diagrams (.svg, .png, .md) representing your project's graph directly generated from the CLI.

skott can be used either via its CLI or JavaScript API. It can either build the project graph using an entrypoint file or build it starting from the current root directory and recursively traverse all directories/folders. Currently, supported files are .js, .jsx, .cjs, .mjs, .ts, .tsx. skott does not rely on module systems for path resolution, it will resolve require/import no matter the configuration.

Getting started

This is a quick start, please check the complete documentation at the skott package level to see more. There are also some examples available there.

Install skott from npm using whatever package manager you like:

<insert-your-package-manager> install skott

Let's see examples of an analysis that will traverse all files (minus the ignored ones via the ignorePattern + .gitignore) starting from the current working directory.

Using the CLI

skott --displayMode=webapp --trackThirdPartyDependencies --ignorePattern="test/**/*"

Using the API

import skott from "skott";

const { getStructure, getWorkspace, findUnusedDependencies, useGraph } = await skott({
    ignorePattern: "test/**/*",
    dependencyTracking: {
        builtin: false,
        thirdParty: true,
        typeOnly: true
    }
});

// Do whatever you want with the generated graph
const { graph, files } = getStructure();
const workspace = getWorkspace();
const unusedDependencies = await findUnusedDependencies();
const { findCircularDependencies, collectFilesDependencies, ...traversalApi } = useGraph();

Note: the API is currently published as ESM only.

Because the graph can become heavy on large codebases, you also have the ability to reduce the scope of the analysis:

Specificying a sub-folder

skott --cwd=packages/skott 

or

import skott from "skott";

const api = await skott({
    cwd: "packages/skott"
});

Specifying an entrypoint

This will strictly traverse the graph starting from the provided entrypoint, discarding all other files not part of that graph.

skott packages/skott/index.ts

or

import skott from "skott";

const api = await skott({
    entrypoint: "packages/skott/index.ts"
});

Why you should use skott or an equivalent project

The whole purpose of skott is to build a graph from your project source code and offer many features relying on that generated graph.

Overall, a generated project graph in software engineering acts as a powerful tool that enhances code comprehension, project management, code maintenance and refactoring, and collaboration. It provides a holistic view of the project's structure, dependencies, and relationships, enabling developers to make informed decisions and streamline their development process.

Moreover, skott aims to provide a comprehensive visual representation of the project's structure, dependencies, and relationships between different components. This visual overview allows developers to better understand the overall architecture and organization of the codebase, making it easier to navigate and identify potential areas of improvement or optimization.

In the essence the main goal of skott is to help developers understand the codebase's structure and to enable more effective refactoring, reducing code duplication, getting rid of circular dependencies and improving overall code quality.

Why you should care about circular dependencies and dead code

  1. Circular (also known as cyclic) dependencies

Let's start with an example of a simple circular dependency between three graph nodes:

In the context of skott, nodes represent JavaScript files.

  graph LR
    src/fileA.js --> src/fileB.js
    src/fileB.js --> src/fileC.js
    src/fileC.js --> src/fileA.js

What is the problem with cycles?

Circular dependencies can make your program crash or introduce inconsistencies while modules are being loaded. Node.js module systems try to resolve circular dependencies using different approaches which are more or less successful. For example, CommonJS can (due to its dynamic nature of resolving imports) introduce inconsistencies when building the project graph.

If you're using ECMAScript modules, you can consider yourself safe about module resolution inconsistencies mentioned above, mostly thanks to its static nature.

Nevertheless, cyclic dependencies at the file-level are sometimes choices but more often code smells revealing design misconceptions, so be sure to double check that.

  1. Dead code

Dead code can be defined as a code literally having no impact on the application, meaning that removing dead code should not alter in any way the behavior of the system. Some module bundlers such as Rollup and Webpack allow to delete some of the dead code leveraging tree shaking.

However, tree shaking is not an easy task and can mostly work with module systems using static-based imports/exports such as ECMAScript modules. To avoid removing code that appears to be used at runtime, module bundlers are being very precise about determining automatically chunks of code that can be safely removed. Module bundlers can also be helped by providing them manually clues about what can be safely removed e.g. /*#__PURE__*/ for Webpack.

If you're not using tools implementing tree shaking, you will be able soon to use skott, which will bring up soon unused imports/exports warnings 🚀

Graph Management

skott is powered by digraph-js, a 0 dependency Node.js library to make Directed Graph construction and traversal effortless.

Parsers

While traversing the project, skott automatically loads the appropriate parser required. When meeting ".js, .cjs, .mjs, .jsx" files, a specific JS parser will be used, otherwise for ".ts, .tsx" files a specific TS parser will be used instead.

  • JavaScript: JavaScript/JSX parsing is done using meriyah
  • TypeScript: TypeScript/TSX parsing is done using typescript-estree

Contributors

Daniel Sadilek
Daniel Sadilek

💻
Salah Eddine Taouririt
Salah Eddine Taouririt

💻
Alexis CHAPPRON
Alexis CHAPPRON

💻
Roberto Yoc
Roberto Yoc

💻
Pedro Lamas
Pedro Lamas

💻
Alexander Khoroshikh
Alexander Khoroshikh

💻

skott's People

Contributors

achp avatar alexandrhoroshih avatar allcontributors[bot] avatar antoine-coulon avatar daniel-relay avatar github-actions[bot] avatar pedrolamas avatar robertoyoc avatar tarrsalah avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

skott's Issues

Support imports map in package.json

Summary

The imports map doesn't seem supported, if I do something like import { t } from "#config/trpc.js"; it doesn't appear in the graph.

Details

Now that we have nodenext in TypeScript, using the imports map is the standard way of aliasing paths internally in a package, so I'd love to have it supported in skott.

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question Answer
Would you consider contributing a PR? Yes

As a side-note, it seems that they are a lot of overlaps between skott and knip. I'm working on a "future-proof" TypeScript monorepo example, and I think skott would be a great addition to it. I'm also already using knip which is already on point to crawl every files and understand dependencies (including imports map), so maybe there could be some kind of collaboration here? 😄

Failed to Resolve with index.tsx files

With the following structure:

  src/common/components/SafeContainer
├── constants.ts
├── elements.tsx
├── index.test.tsx
└── index.tsx

Then in another file I have:

import { SafeContainer } from "@/common/components/SafeContainer";

This is my tsconfig for the aliasing:

{
  "compilerOptions": {
    "baseUrl": ".",
    "jsx": "react",
    "esModuleInterop": true,
    "paths": {
      "@/*": ["./src/*"],
      "@assets/*": ["./assets/*"]
    },
    "strict": true
  },
  "exclude": ["node_modules", "dist"]
}

After running skott index.js --verbose I see

Start resolution for src/common/components/SafeContainer 
 ✖ Failed to resolve src/common/components/SafeContainer/index.js 

 ✖ Failed to resolve src/common/components/SafeContainer.ts 

 ✖ Failed to resolve src/common/components/SafeContainer/index.ts 

 ✖ Failed to resolve s.ts 

 ✖ Failed to resolve src/common/components/SafeContainer.js 

 ✖ Failed to resolve src/common/components/SafeContainer.tsx 

 ✖ Failed to resolve src/common/components/SafeContainer.jsx 

 ✖ Failed to resolve src/common/components/SafeContainer/index.js 
 

Looks like library is not checking index.tsx/index.jsx paths?
Thanks for your help.

Customize Graph layout and rendering to better fit with various project shapes and graph sizes

Currently, the web app is not really customizable and takes opinionated choices using heuristics to know how to dispose all the graph nodes in the 2D space. However having heuristics that fit well for all types of graphs is not an easy task because it is nearly impossible to find a universal disposal pattern that fits well for all types of project architectures:

  • graphs sizes can vary between few nodes to thousands of nodes
  • graphs shapes can be very different from one architecture to another architecture. For example some layout might only make sense in the context of hexagonal architecture.
  • graphs can include circular dependencies or deeply nested dependencies that make hierarchical layout impossible to use.

One thing that could solve in a universal way that problem would be for to let the user select the type of layout that fits the best for its project.

This would probably require a rework of the sidebar to provide better UI/UX.

Feature request: Distinguish safely-hoistable circular imports from those that will fail at runtime

If you're building a recursive structure like a DOM or a type system (in this case the latter), at some point you'll have to choose to either maintain a massive source file containing all your nodes or to judiciously allow some circular imports. The situation is summarized in more detail in a comment here.

Unfortunately, I'm not aware of any tools that will automatically detect when circular imports will be problematic at runtime, and the manual debugging process is painful if I make a mistake.

I'm not sure how viable this would be to add as a feature to skott, but if possible, it would single-handedly be justification for adopting it in projects like mine!

Add support for `npx skott`

The CLI doesn't seem to work when run via npx:

Screen Shot 3

If it's available globally, then npx will also work, so make sure you uninstall the global version before attempting this.

Support ignore patterns to prevent files from being included in the emitted graph

Thanks again to @pigoz for that other feature request.

The issue #33 will deal with node filtering directly in the webapp meaning that the graph has internally all the files.

However for other display modes that are not dynamic, we might want to have the same feature which is preventing some specific files to be included in the emitted graph. As of now, skott internally has already a set of files to ignore but it's opinionated and not flexible nor configurable.

Consequently the idea would be to provide an ignorePatterns option discarding a set of files from the analysis. This feature could be seemlessly combined with the webapp node filter feature.

skott --ignorePattern "/**/config/*"

Support JSX/TSX

To be fully compatible with most of the projects used in the ecosystem, skott must support JSX/TSX parsing.

Tool crashes when run (ERR_INVALID_ARG_TYPE)

For starters, this tool hangs indefinitely when run in our repo. We discovered that for some reason, we have to delete a .docusaurus directory from one of our packages to get this tool to finish at all. After doing that, the tool crashes when run:

$ rush graph


Rush Multi-Project Build Tool 5.93.2 - https://rushjs.io/
Node.js version is 16.19.1 (LTS)


Trying to acquire lock for pnpm-7.29.3
Acquired lock for pnpm-7.29.3
Found pnpm version 7.29.3 in C:\Users\kgetz\.rush\node-v16.19.1\pnpm-7.29.3

Symlinking "C:\Users\kgetz\Work\event-viewer\common\temp\pnpm-local"
  --> "C:\Users\kgetz\.rush\node-v16.19.1\pnpm-7.29.3"
Acquiring lock for "common\autoinstallers\rush-plugins" folder...
Autoinstaller folder is already up to date

Starting "rush graph"

✓ Rush configuration found
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at new NodeError (node:internal/errors:387:5)
    at validateString (node:internal/validators:162:11)
    at Object.join (node:path:429:7)
    at resolveAliasToRelativePath (file:///C:/Users/kgetz/Work/event-viewer/common/autoinstallers/rush-plugins/node_modules/.pnpm/[email protected]/node_modules/skott/dist/src/modules/walkers/ecmascript/typescript/path-alias.js:37:36)
    at resolvePathAlias (file:///C:/Users/kgetz/Work/event-viewer/common/autoinstallers/rush-plugins/node_modules/.pnpm/[email protected]/node_modules/skott/dist/src/modules/walkers/ecmascript/typescript/path-alias.js:51:16)
    at EcmaScriptDependencyResolver.resolve (file:///C:/Users/kgetz/Work/event-viewer/common/autoinstallers/rush-plugins/node_modules/.pnpm/[email protected]/node_modules/skott/dist/src/modules/resolvers/ecmascript/resolver.js:81:40)
    at Skott.collectModuleDeclarations (file:///C:/Users/kgetz/Work/event-viewer/common/autoinstallers/rush-plugins/node_modules/.pnpm/[email protected]/node_modules/skott/dist/src/skott.js:182:47)
    at async Skott.buildFromRootDirectory (file:///C:/Users/kgetz/Work/event-viewer/common/autoinstallers/rush-plugins/node_modules/.pnpm/[email protected]/node_modules/skott/dist/src/skott.js:262:13)
    at async Skott.initialize (file:///C:/Users/kgetz/Work/event-viewer/common/autoinstallers/rush-plugins/node_modules/.pnpm/[email protected]/node_modules/skott/dist/src/skott.js:270:13)
    at async skott (file:///C:/Users/kgetz/Work/event-viewer/common/autoinstallers/rush-plugins/node_modules/.pnpm/[email protected]/node_modules/skott/dist/index.js:8:27) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Restore behavior where we can provide falsy values to boolean options with the CLI

Regression of the CLI introduced in #88

Previously with sade, we were able to provide to boolean options both truthy and falsy values:

skott --trackTypeOnlyDependencies=false # works
skott --trackTypeOnlyDependencies=true # works

Using commander, that behavior is no longer supported

skott --trackTypeOnlyDependencies=false # does not work

When the option default value is "false" that's fine. But when the default value is "true" that's problematic. For instance trackTypeOnlyDependencies is one option whose default value is "true".

skott --trackTypeOnlyDependencies # we can only enable it, even though its default value is already true
```ou 

With commander it seems that you can bypass that behavior using `--no-<command-name>` allowing you to provide falsy values.

Disable graph out when using Skott as a command line tool

Hi everyone,

I'm not logging an issue this is more a general question about the usage of Skott.

So I'm looking at switching from Madge to Skott as it comes with a lot more out of the box and has a better api, however, I have noticed that when using Skott it outputs a visual graph whether that is via the web app or some external file.

I'm wanting to use Skott purely as a command line tool, Is it possible to disable the visual output and instead just have some basic information outputted to the console e.g. if I have a circular dependency show me information about that.

Any feedback and comments are appreciated.

Thank's,

Alex

Add `usage` point in README.md

Please provide better documentation for usage.

It should tell how to install and use the tool.
Because now it's not clear for me.

NPM workspaces considered third party packages

Hi,

It seems like if you are using npm workspaces you can't build a graph beyond/through those package boundaries.

I wonder if it is possible to either:

  1. deeply graph third party packages, or
  2. distinguish npm workspace packages from third party packages.

Thanks for all your effort on this project, it looks great!

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined

I have the latest version of this tool installed, but it crashes every time I run it:

$ skott --version
skott, 0.28.0

 Skott exited with code 0

$ skott -s

 Running Skott from current directory: event-viewer
⠇ Initializing Skott
 Skott exited with code 1
node:internal/errors:490
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at new NodeError (node:internal/errors:399:5)
    at validateString (node:internal/validators:163:11)
    at Object.join (node:path:429:7)
    at resolveAliasToRelativePath (file:///C:/Users/kgetz/AppData/Roaming/nvm/v18.16.0/node_modules/skott/dist/src/modules/walkers/ecmascript/typescript/path-alias.js:10:36)
    at resolvePathAlias (file:///C:/Users/kgetz/AppData/Roaming/nvm/v18.16.0/node_modules/skott/dist/src/modules/walkers/ecmascript/typescript/path-alias.js:88:35)
    at EcmaScriptDependencyResolver.resolve (file:///C:/Users/kgetz/AppData/Roaming/nvm/v18.16.0/node_modules/skott/dist/src/modules/resolvers/ecmascript/resolver.js:113:40)
    at Skott.collectModuleDeclarations (file:///C:/Users/kgetz/AppData/Roaming/nvm/v18.16.0/node_modules/skott/dist/src/skott.js:213:47)
    at async Skott.buildFromRootDirectory (file:///C:/Users/kgetz/AppData/Roaming/nvm/v18.16.0/node_modules/skott/dist/src/skott.js:285:13)
    at async Skott.initialize (file:///C:/Users/kgetz/AppData/Roaming/nvm/v18.16.0/node_modules/skott/dist/src/skott.js:305:13)
    at async skott (file:///C:/Users/kgetz/AppData/Roaming/nvm/v18.16.0/node_modules/skott/dist/index.js:27:27) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Node.js v18.16.0

Multiple Ignore Patterns

I have a few different patterns I want to ignore via --ignorePatterns but it seems this can only accept one pattern? It would be great if this could accept an array similar to --fileExtensions

Allow Skott to perform a global analysis without specifying an entrypoint

Following #2 #4 which were both resolved in #3, the only missing piece to use Skott in most cases is the ability to perform a global analysis without specifying any entrypoint.

The reason for that is that some projects such as Next.js use file-system based routing which implicitly create links between files (without using any import/export statement, thing that Skott relies on to create the graph).

Consequently the goal is to be able to deeply collect a specific set of independent files (basically .ts, .js, .tsx, .jsx) which are not linked by any import/export statement. This would allow Skott to produce sub graphs and unify them as a single project graph.

In the context of monorepo/workspaces, this could also be useful.
Given two independent libs that are evolving independently (without any dependency toward each other):

|__lib-a/ 
           a.js
           root-a.js
|__lib-b/
           b.js
           root-b.js

The resulting graph could be:

graph TB;

a.js --> root-a.js
b.js --> root-b.js

Does not include .tsx deps when extension in import statement

Thanks for the tool - I'm trying to run this over a react/typescript project with many .tsx dependencies.

Unfortuantely it seems the tool misses out when we import a tsx file without specifying the extension.

E.g. we have a file called SurveyVideo.tsx, and another file that imports it:

// Does not include SurveyVideo.tsx in the dep graph 
import * from "./SurveyVideo";
// Works
import * from "./SurveyVideo.tsx";

Can we get around this with the current config or is this a bug?

RFC: revamp skott web application

Started in #72

The goal is to rebuild the application from scratch to add a lot of features and DX improvements, but also enhance maintainability and reduce complexity.

The Graph network would still remain the central piece of the application, but a lot of data collected by skott is still not leveraged.

Currently, I'm thinking of adding 6 sections:

  • Summary section: visualize global stats (as it was done in the previous version but in a better way) + listing npm (+ show unused dependencies) & node.js dependencies (if flags were enabled)
  • File Explorer section: visualize files à la VS Code using a file explorer that would allow you to browse files, filter files and interact with them (focus on file, isolate file, etc)
  • Graph Configuration section: configuring the graph, changing the layout mode allowing to draw graphs differently based on user choice (hexagonal architecture -> hierarchical layout) as it's too complex to do it automatically. But also changing nodes colors, nodes spacing, etc.
  • Dependencies Explorer section: section focused on displaying dependencies of the graph either they are internal (cycles), external (third-party) or builtin (native to the Node.js platform).
  • Cycles Explorer section: section focused on cyclic dependencies that allows you to filter out cycles and better see what's the composition of each cycle, using the Network only we can't distinguish sub-cycles
  • Interactive Playground section: focused on graph traversal, filtering dependencies starting from nodes, etc. This would allow a user to say isolate one file and draw me all it's deep dependencies or isolate one file and draw me all files that depend on it.

Here is a preview of the UI:

Capture d’écran 2023-07-20 à 12 01 32

Turn various display modes into Skott plugins to avoid embedding everything in the core by default

Currently, all the display modes are builtin into skott so that you just have to install skott to use all of them. The problem with that user-friendly approach is that it loads most of the time unnecessary modules both during the module resolution and during the install.

In the same way as for the webapp which is already an external package (but no dynamically loaded yet), all other external display modes should follow the same pattern.

Skott own modules are not installed

Hi,
Thanks for your work.
I've installed your package globally but I get an error

Error

➜  sgo ✗ skott src/App.tsx

 Running Skott from entrypoint: src/App.tsx
 Processed 1 file (2.285ms)
 ✓ no circular dependencies found (depth=Infinity)
 Third-party dependencies: ❌
 --> Use --trackThirdPartyDependencies to enable tracking.
 Built-in dependencies: ❌
 --> Use --trackBuiltinDependencies to enable tracking.
 Type-level only dependencies: ✅

 Skott exited with code 1
node:fs:1438
  handleErrorFromBinding(ctx);
  ^

Error: ENOENT: no such file or directory, scandir '/Users/aze/.config/yarn/global/node_modules/skott/node_modules/skott-webapp/dist'
    at readdirSync (node:fs:1438:3)
    at totalist (/Users/aze/.config/yarn/global/node_modules/totalist/sync/index.js:6:12)
    at module.exports (/Users/aze/.config/yarn/global/node_modules/sirv/build.js:149:3)
    at openWebApplication (file:///Users/aze/.config/yarn/global/node_modules/skott/dist/bin/ui/webapp.js:45:20)
    at displaySkott (file:///Users/aez/.config/yarn/global/node_modules/skott/dist/bin/main.js:134:9) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/Users/aze/.config/yarn/global/node_modules/skott/node_modules/skott-webapp/dist'
}

I checked here /Users/aze/.config/yarn/global/node_modules/skott and found out that packages are not installed.
Tried to install them myself using yarn installbut I end up on a maybe encrypted variable @skott%2fconfig

Yarn-error.log

Arguments: 
  /Users/aze/.nvm/versions/node/v16.19.0/bin/node /opt/homebrew/Cellar/yarn/1.22.19/libexec/bin/yarn.js install

PATH: 
  /Users/aze/.nvm/versions/node/v16.19.0/bin:/Users/aze/.rbenv/shims:/Users/aze/Library/Caches/fnm_multishells/1122_1671351776860/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin:/Users/aze/Library/Android/sdk/emulator:/Users/aze/Library/Android/sdk/platform-tools:/Users/aze/Library/Android/sdk/tools:/Users/aze/Library/Android/sdk/tools/bin

Yarn version: 
  1.22.19

Node version: 
  16.19.0

Platform: 
  darwin arm64

Trace: 
  Error: https://registry.yarnpkg.com/@skott%2fconfig: Not found
      at Request.params.callback [as _callback] (/opt/homebrew/Cellar/yarn/1.22.19/libexec/lib/cli.js:66145:18)
      at Request.self.callback (/opt/homebrew/Cellar/yarn/1.22.19/libexec/lib/cli.js:140890:22)
      at Request.emit (node:events:513:28)
      at Request.<anonymous> (/opt/homebrew/Cellar/yarn/1.22.19/libexec/lib/cli.js:141862:10)
      at Request.emit (node:events:513:28)
      at IncomingMessage.<anonymous> (/opt/homebrew/Cellar/yarn/1.22.19/libexec/lib/cli.js:141784:12)
      at Object.onceWrapper (node:events:627:28)
      at IncomingMessage.emit (node:events:525:35)
      at endReadableNT (node:internal/streams/readable:1358:12)
      at processTicksAndRejections (node:internal/process/task_queues:83:21)

npm manifest: 
  {
    "name": "skott",
    "version": "0.12.1",
    "description": "Automatically construct and visualize Graphs generated from your JavaScript and TypeScript project",
    "author": "Antoine Coulon",
    "license": "MIT",
    "repository": {
      "type": "git",
      "url": "git+https://github.com/antoine-coulon/skott.git"
    },
    "bugs": {
      "url": "https://github.com/antoine-coulon/skott/issues"
    },
    "main": "dist/index.js",
    "bin": {
      "skott": "./dist/bin/cli.js"
    },
    "type": "module",
    "exports": {
      ".": "./dist/index.js"
    },
    "types": "./dist/index.d.ts",
    "dependencies": {
      "@typescript-eslint/typescript-estree": "^5.40.0",
      "compression": "^1.7.4",
      "digraph-js": "^2.0.2",
      "estree-walker": "^3.0.1",
      "fs-tree-structure": "0.0.1",
      "kleur": "^4.1.5",
      "ligie": "^0.0.6",
      "meriyah": "^4.2.1",
      "openurl": "^1.1.1",
      "ora": "^6.1.2",
      "polka": "^0.5.2",
      "sade": "^1.8.1",
      "sirv": "^2.0.2",
      "skott-webapp": "0.0.3",
      "table": "^6.8.0"
    },
    "devDependencies": {
      "@nodesecure/eslint-config": "^1.3.0",
      "@skott/config": "0.0.1",
      "@types/chai": "^4.3.0",
      "@types/compression": "^1.7.2",
      "@types/mocha": "^9.0.0",
      "@types/node": "^16.11.12",
      "@types/polka": "^0.5.4",
      "@types/sade": "^1.7.4",
      "chai": "^4.3.4",
      "eslint": "^8.4.1",
      "eslint-config-prettier": "^8.3.0",
      "eslint-import-resolver-typescript": "^2.5.0",
      "eslint-plugin-import": "^2.26.0",
      "eslint-plugin-prettier": "^4.0.0",
      "memfs": "^3.4.6",
      "mocha": "^9.1.4",
      "prettier": "^2.5.1",
      "rimraf": "^3.0.2",
      "ts-node": "^10.8.1",
      "typescript": "^4.5.2"
    },
    "files": [
      "dist",
      "./README.md"
    ],
    "keywords": [
      "javascript",
      "graph",
      "nodejs",
      "visualization",
      "module",
      "dependencies"
    ],
    "scripts": {
      "build": "rimraf ./dist && tsc --project tsconfig.build.json",
      "lint": "eslint .",
      "prestart": "pnpm run build",
      "start": "node dist/bin/cli.js",
      "test": "mocha --parallel"
    }
  }

yarn manifest: 
  No manifest

Lockfile: 
  No lockfile

Any idea how to solve this ?

webapp display mode throws error in wsl

Running in WSL (Windows Subsystem Linux) throws an error when trying to open the webapp URL:

$ skott --displayMode=webapp

 Running Skott from current directory: -

 Processed 183 files (1885.137ms)

 ✓ no circular dependencies found (depth=Infinity)

 Third-party dependencies: ❌

 --> Use --trackThirdPartyDependencies to enable tracking.

 Built-in dependencies: ❌

 --> Use --trackBuiltinDependencies to enable tracking.

 Type-level only dependencies: ✅

 Prefetching data... 

 💻 Opened skott webapp on http://127.0.0.1:41707

 Skott exited with code 1
node:events:495
      throw er; // Unhandled 'error' event
      ^

Error: spawn xdg-open ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:284:19)
    at onErrorNT (node:internal/child_process:477:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:290:12)
    at onErrorNT (node:internal/child_process:477:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn xdg-open',
  path: 'xdg-open',
  spawnargs: [ 'http://127.0.0.1:41707' ]
}

Node.js v18.18.0

Feature request: group modules in visualizaton using project structure

Hello and thanks for a great library!

Summary

Context

Our project is using approach similiar to Feature Sliced Design, where code is divided into layers of some entities, all of which have some sort of public API

Example:

/src
 /features <-- some "layer" of the project architecture
  /feature-a
   index.ts <-- public API of the feature
   lib.ts
   utils.ts
   etc.ts <-- various internals, that are hidden behind public API
  /feature-b
   index.ts

Problem

But if i run skott against project entrypoint or any of the features index file - the generated visualization is very noisy, because all of the internal modules are exposed right away

Suggestion

Since all of these features, widgets, etc are essentialy "groups" of modules behind single entrypoint with a public API, i suggest to add an option to provide glob-like pattern, which would describe them:

const { ... } = await skott({
  entrypoint: "src/index.ts",
  groups: {
   features: "src/features/{name}/*",
   widgets: "src/widgets/{name}/*",
  }

The visualization then can use these groups, e.g. to skip the render of each and every internal module of src/features/feature-a and render it as the single features/feature-a node

This should make the visualization:

  • much easier to read, since internal modules are hidden
  • faster to render, since there would be less graph nodes to handle

Details

This feature would also allow to use skott with bigger projects.

For example our project now has about 7000 modules in it and currently skott just can't handle it in any way, since web-app graph takes forever to load and svg export shows just something like "Maximum size exceeded"

Visualization does work, if i choose an src/widgets/{name}/index.ts as an entrypoint, but result is still not really useful because of all of the internals being exposed:

image

At the same time our project has only about a 200-300 such "services", "features" and "widgets", which graph rendering should handle much easier

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question Answer
Would you consider contributing a PR? Yes

Exported namespaces from third-party dependencies are reported as a specific dependency instead of being part of the same one

skott can track third-party dependencies using the --trackThirdPartyDependencies option, and I currently noticed a problem where npm dependencies were reported more than once where accessing namespaces from them:

@effect-ts/core/Effect and @effect-ts/core/Effect/Layer should be part of the same dependency which is indeed @effect-ts/core. Currently, both are reported as distinct dependencies.

imports misclassified as third party dependencies

Hi!

It seems that if I have a relative path in my import such as:
import myObj from './other/folder'
it gets correctly tracked as a dependency but if I have a absolute path import that relies on the tsconfig "baseUrl" such as:
import myObj from 'absolute/path/folder'
it gets classified as a third party dependency.

I ran the command using the cli and have specified my tsconfig file using --tsconfig

skott installation adds 544 MB to node_modules

Hello! First of all, thanks for this fantastic project! Skott has been incredibly useful for me.

However, I've noticed an issue with the node_modules directory that Skott's installation generates – it's incredibly large. A fresh installation comes in at a whopping 544 MB in dependencies! This seems like quite a hefty price to pay to have Skott as a devDependency for my project.

npm install skott
Screenshot 2023-10-03 at 01 25 37

I've come across some related issues that I'd like to discuss:

  1. I believe TypeScript should be placed in devDependencies instead.
  2. There are other packages that could potentially be moved to devDependencies as well.
  3. The @tabler package takes up a significant amount of space due to its icons, which are likely unused.
  4. It might be worth considering bundling the package before publishing it to npm. This could result in substantial improvements for Skott, potentially reducing its size by a factor of 10x to 25x.

I hope we can address these concerns to make Skott even more efficient and developer-friendly

Make incremental comparison consistent for all types of project structure

Since v0.12.1, incremental mode was enabled by default. It turns out that this works well for JavaScript/TypeScript standard configurations but lacks off consistency when being used with more advanced configurations (specifically TypeScript deep path aliases #20) and TypeScript/JavaScript edge cases of path resolutions.

From now on with #21 until a fix (will be resolved with this issue), incremental will be disabled by default (can still be used with the flag incremental: true).

This feature will eventually be enabled by default once we find it stable enough given all feedbacks across all types of projects.

Skott does not read typescript path aliases if entrypoint is not a TS file

Context

I tried s skott on a React-Native project where almost all files are using Typescript except for the entry file.
And I noticed that all my path aliases were considered as "third party module". Even with the --tsconfig="./tsconfig.json" argument.

How to reproduce

  • Init a 100% Typescript project and setup path aliases:
  • Run skott, it should works.
  • Now update to entrypoint to use .js file extension
  • Now path aliases resolution does not work anymore.

That's because if the entry point is not a TS file, the TS configuration is not loaded
See here:

https://github.com/antoine-coulon/skott/blame/d18c1f800276752431d0177597d1fa4f86db5938/packages/skott/src/skott.ts#L494-L502

Suggestion

If the user use tsconfig as argument for the cli, skott should consider that the project is using typescript

Publish a set of benchmarks after each release with a various set of projects

Recently, a lot of additions/refactoring have been going on and I have not been following performance impacts it had overall. Even is Skott is not designed to be super performant, it still has to be fast enough as it aims to bring a good DX.

To track that, it could be valuable to process benchmarks on the same set of projects (from small to big ones) and expose them publicly as done here in digraph-js after each release.

Invalid Skott config for 0.16.x versions

Windows 10
Node.js v18.13.0
Skott v0.15.3 (running as: [ ] npx, [x] global, [ ] local)

PS D:\Projects\.........> skott .\src\app.ts

 Running Skott from entrypoint: .\src\app.ts
⠋ Initializing Skott
 Skott exited with code 1
file:///C:/Users/YoYurec/AppData/Roaming/npm/node_modules/skott/dist/src/config.js:30
        throw new Error(`Invalid Skott config. Reason: ${D.draw(decodeError)}`);
              ^

Error: Invalid Skott config. Reason: required property "dependencyResolvers"
└─ cannot decode undefined, should be Array<unknown>
    at file:///C:/Users/YoYurec/AppData/Roaming/npm/node_modules/skott/dist/src/config.js:30:15
    at C:\Users\YoYurec\AppData\Roaming\npm\node_modules\skott\node_modules\fp-ts\lib\Either.js:166:49
    at pipe (file:///C:/Users/YoYurec/AppData/Roaming/npm/node_modules/skott/node_modules/@fp-ts/data/_mjs/Function.mjs:192:14)
    at makeSkottConfig (file:///C:/Users/YoYurec/AppData/Roaming/npm/node_modules/skott/dist/src/config.js:29:12)
    at skott (file:///C:/Users/YoYurec/AppData/Roaming/npm/node_modules/skott/dist/index.js:7:43)
    at displaySkott (file:///C:/Users/YoYurec/AppData/Roaming/npm/node_modules/skott/dist/bin/main.js:96:33)
    at o.parse (C:\Users\YoYurec\AppData\Roaming\npm\node_modules\skott\node_modules\sade\lib\index.js:1:3402)
    at file:///C:/Users/YoYurec/AppData/Roaming/npm/node_modules/skott/dist/bin/cli.js:41:5
    at ModuleJob.run (node:internal/modules/esm/module_job:194:25)

have no idea why (((

Error in Jest Environment

I was trying to use skott in a jest environment in order to create a unit test that scans my directories for circular dependencies, but I came across the following error:

({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){import * as O from "@effect/data/Option";

This issue is probably caused by the usage of ECMAScript Modules in skott. Is there a chance of backward compatibility being implemented into skott, so there won't be any issues like that with other libraries as well?

Webapp display mode node spacing

Hi,
I'm trying Skott out on a pretty big codebase - looks great so far, but on the webapp display many nodes are layed out one on top of the other.
Is there any parameter that can determine minimal node spacing?
image

TypeScript path aliases are not resolved when the base tsconfig does not specify path aliases

Thanks to @pigoz for reporting that issue!

The issue is related to TypeScript path alias resolution when having multiple tsconfig.json with some inheritance described as such:

tsconfig.json

{
  "extends": "./tsconfig.base.json"
}

tsconfig.base.json

{
  "compilerOptions": {
       "paths": {}
  }
}

In the case where tsconfig.json specifies some paths there is no problem merging all the paths found from both tsconfig and tsconfig.base.

However, it seems that when only the extended (in that example tsconfig.base.json) config file specifies "paths", Skott is not able to resolve any paths at all.

Automatically ignore files and directories described in .gitignore

As of now, skott only ignores a pre-defined set of directories.

However, this does not prevent skott from analyzing files/directories including JS/TS files that are not directly part of the source code (bundles, artifacts, etc).

This creates unnecessary overhead and might even cause runtime defects when encountering generated JavaScript/TypeScript files not meant to be analyzed.

The goal is to make skott able to deeply find .gitignore files as a pre-analysis step and discard them from the file tree traversal.

Dual publish ESM/CJS

skott is only published as ESM as of now.

Until now people have been mostly using skott through the CLI so CJS/ESM was not a thing but for the API it would be more convenient to have a dual publish.

Even though I usually advocate the use of ESM-only packages, CJS is still a thing and smooth transitions should still be possible for everyone.

UX - Webapp display - search is difficult

I really like your tool,

Therefore I am making suggestions to see it evolve.

Once the graph is displayed, like this :
Capture d’écran 2022-12-19 à 18 02 15

It would be great to add these features :

  • underline the specified file as parameter
    • with command for example skott src/pages/PaymentPage/PaymentPage.component.tsx, I would like to see the item PaymentPage.component.tsx highlighted
  • allow browser text search (cmd + F)
    • OR create your own search and highlight searched item

By the way I would be pleased to help you on this.

Let me know if you find my suggestions useful,

You can contact me on : [email protected]

FeatureRequest | Typescript support

Feature Request

Will skott support typescript projects?

Our use case is an Nx monorepo with Nextjs as the main app with several react libraries.
Thanks for this interesting library!

View unused modules in a monorepo for dead code elimination

I would like to see unused modules so that I can easily remove dead code from the codebase.

Note:
Since the module can be used in NOT production code like tests, storybook and other tooling files, there should be an option to skip them

It doesn't seem to correctly find aliases in tsconfig

You can see my config at https://github.com/pixiebrix/pixiebrix-extension/blob/ae55fca9e0c610c801f05107f2cde021db09ef0f/tsconfig.json#L19-L26

I'm using a similar config in Madge and I think that one works: https://github.com/pixiebrix/pixiebrix-extension/blob/ae55fca9e0c610c801f05107f2cde021db09ef0f/package.json#L22

 npx skott src/pageScript/pageScript.ts --trackBuiltinDependencies --trackThirdPartyDependencies --displayMode=file-tree

 Running Skott from entrypoint: src/pageScript/pageScript.ts

 Processed 1 file (13.471ms)

 ✓ no circular dependencies found (depth=Infinity)

  pageScript.ts

 npm third-party dependencies: 

  @/contentScript (imports=1)
  @/frameworks (imports=1)
  @/messaging (imports=1)
  @/pageScript (imports=1)
  @/runtime (imports=1)
  @/types (imports=1)
  @/utils (imports=1)
  @/vendors (imports=1)
  lodash (imports=1)

 ✓ no builtin dependencies found

 Skott exited with code 0

Path Aliases only work for typescript projects

js projects can have aliases configured in a jsconfig file.

Currently they won't be discovered if you run the CLI with --tsconfig jsconfig.json, however.

This looks to be because buildFromEntrypoint in packages/skott/src/skott.ts only runs buildPathAliases for .ts entrypoints:

if (isTypeScriptModule(entrypointModulePath)) {
  const rootTSConfig = await buildPathAliases(
    this.fileReader,
    this.config.tsConfigPath,
    this.#workspaceConfiguration.pathAliases,
    this.logger
  );
  this.#workspaceConfiguration.typescript = rootTSConfig;
}

Thanks again for the great work! 🙏

Improve web app customization by allowing graph nodes to be filtered and compacted

Kudos to @pigoz for suggesting all these UI improvements. His great effect-sql project will be used in as an example in the following screen shots.

The goal of this feature request is to be able to customize the visualization of graph elements rendered such as

  • compact directories to a single node during visualization
  • be able to filter nodes that are displayed on screen (implemented in skott webapp v2)

1) compact directories to a single node during visualization

Skott generates a graph node per file. It would be great to be able to compact all the files within a specific directory/sub directory. For example on the screen shot just above, all the files included in src/builders would become a single node such as src/builders/* instead of represent each file by its own node.

effect-sql

Would become compacted into something like:

effect-sql-2

I think this would also need the UI to be more explicit towards nodes by making the distinction between folders and files, maybe by using icons or colors?

2) Filter graph nodes

Allowing nodes to be displayed/not displayed could also be an useful feature because Skott sometimes add nodes to the graph that are not valuable to visualize (module bundlers and test configs for instance).

I think this could be done via a CLI/API option from Skott by providing a ignore files glob pattern to also allow other display modes to benefit from that but most of the time I think it makes more sense for now to only hide some nodes dynamically depending on the analysis context (refactoring, architectural/design views). In the end, I think both will be possible.

Both those features need UI/UX thinking. My first thought would be to provide those features from the sidebar.

Skott webapp - Allow cycles to be fetched in a non-blocking way

Currently, once the webapp opens up (when using --displayMode=webapp) all the data generated by the analysis is pre-fetched, including cyclic dependencies, which can be expensive hence blocking the graph from being displayed while computing the cycles.

Given that we don't render the cycles on the first paint, it would be useful to compute and fetch the circular dependencies only once we decide to display them (and not while rendering for the first time).

It could be provided via a new endpoint /cycles triggered once the display of this data is requested from the UI.

Allow "import { type X }" to be part of the "typeOnly" tracking process

Currently, there is the --trackTypeOnlyDependencies CLI flag and dependencyTracking: { typeOnly: boolean } API option both supposed to turn on/off the type-level dependency tracking.

However, this only supports TypeScript import type { X } while type-level imports can also be expressed as import { type X }.

Goal of the issue: discard modules imported using import { type X } from the graph when the config says so.

Enable third-party tsconfig extension loading

Currently when some TypeScript configuration file extends another tsconfig.json from a relative location, everything should be fine.

However some extended configuration comes from the node_modules ("extends": "@skott/config") skott won't be able to load that additional config as of now.

First solution that comes in mind: use createRequire to dynamically resolve the JSON module to the appropriate location.

Exclude TS type from circular detection

I think Typescript types should be excluded from circular dependencies detection

All type declarations will be deleted when compiling ts file, so there will be no negative effect in final code

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.