Loading the RAR format and parsing a RAR5 file (even with an empty file, like this sample) will crash the Chrome tab renderer, because eating too much memory probably.

crash.rar.zip (because github)

As a workaround first at least do not auto-parse at startup if the previous parse crashed the browser (ie. the previous parse started but never finished).

As there is no guarantee that these fields will be supported in the future.

Candidate lib: https://github.com/APIs-guru/graphql-voyager

• EnsureContents throws exception on error and the original value of the field is not available (only in the exception), but the erroneous node cannot be visualized and selected this way
• Getters throw exception and the parsed value is not available

Currently we use a non-standard font and the WebIDE looks slightly different on non-Windows based systems.

When reverse-engineering a format you often need to put a structure on some offset. That offset can be determined by other members offsets and sizes. So I propose to add a feature with the following look&feel:
1 a user selects the size or pos of the field to be modified and the depth to which changes will be propagated. Webide goes up n levels of structs (0 - the struct where the chosen field in, 1 the structure containing the structure where the field is in), cuts the piece of binary, cuts the needex pieces of ksy, moves it into a separate tab. This will allow to experiment solely on that piece without screwing the rest of parsed binary.
2 user moves the structures after (in the cace of size) or in (in the case of pos) that field with mouse and/or keyboard. Moving is counted as offset to the initial position determined by ksy file. When user feels that the position is correct, he stops, WebIde modifies ksy.
3 then modified ksy can be merged back to the ksy it was ripped from.

Also it may be useful to show the bytes not as blocks with hex as colorful blocks , and structures as colorful frames around this blocks. This may allow easier spotting of patterns in data.

A few people already mentioned that: why can't we work with something tree-like in Web IDE, instead of just a text editor with a YAML file?

Instead of editing (and actually re-parsing YAML on every occassion), we could provide tree-like UI and have some basic operations, like "Add / Edit Attribute", "Add Type", "Rename / Move Type", etc, with some wizard-like pop-up windows. Tehcnically, YAML is already a tree, and we already have a tree UI library used for tree representation of the data, so why don't we reuse it?

It turns out that there are many more timestamp formats in the wild besides UNIX timestamp. It would be cool to show more of them in "converter" to aid timestamp detection guesswork.

Timestamps

Continuous interval measurements

• Unix timestamp

  • Size: u4le/be
  • Definition: seconds since 1970-01-01 00:00 UTC

• Microsoft FILETIME

  • AKA:
    • Windows NT time format
    • NTFS file time
    • LDAP time
    • Active Directory time
  • Size: u8le
  • Definition: 100-nanoseconds intervals since 1601-01-01 00:00 UTC

• WebKit/Chrome timestamp

  • Size: u8 (?)
  • Definition: microseconds since 1601-01-01 00:00 UTC

• Apple Cocoa Core Data timestamp

  • AKA:
    • Mac absolute time
  • Size: ?
  • Definition: seconds since 2001-01-01 00:00 UTC

• Apple Mac OS X HFS+ timestamp

  • AKA:
    • Mac timestamp
  • Definition: seconds since 1904-01-01 00:00 UTC

• Seconds since year 0

  • AKA:
    • MySQL time
  • Size: u8 (?)
  • Definition: seconds since

• Excel timestamp

  • AKA:
    • Microsoft timestamp
  • Size: f8
  • Definition: days since 1899-12-31

Complex structures

• Microsoft FAT date time

  • Size: 4 byte packed structure = 2 bytes for date + 2 bytes time
  • Timezone dependent
  • Resolution: 2 sec

• Microsoft SYSTEMTIME

  • Size: 16 byte structure
  • Definition: individual fields for components
  • Timezone resolution: ?
  • Resolution: ?

• ISO9660 decimal timestamp

  • Size: 17 bytes structure
  • Definition: individual fields for components, mostly ASCII strings
  • Resolution: 0.01 sec
  • Timezone resolution: 15 minutes

• ISO9660 binary timestamp

  • Size: 7 bytes structure
  • Definition: individual fields for components
  • Resolution: 1 sec
  • Timezone resolution: 15 minutes

I don't know exact details, but probably parsing a complex file can use all the allowed stack up in Chrome.

(@xavrr mentioned the issue on Gitter: "is there a way to increase the stack size for chrome ? to allow for bigger size parsing ?")

I'd like to propose a "watch" functionality. It should probably work like that:

• User selects a particular node in a tree, and calls "Add watch" on it
• This tree path is encoded into some JavaScript expression, like foo.bar.someArray[42].someOtherElement - and user can edit it
• Afterwards, an additional window with a list of watch expressions appears, which always shows exact result of evaluation of this expression, no matter what happens with opened/closed states of the tree, no matter what happens with the .ksy source, etc.
• If there are multiple open data files, "watch expression" should show evaluation result for every one of them.

Bonus:

• If we'd ever allow opening many (i.e. dozens or hundreds of) files, then it would be cool to show some frequency statistics (i.e. value evaluation result 7 is encountered 15 times, result 1 is encountered 4 times, result 5 if encountered twice, etc).

About box has a link to https://ide.kaitai.io/LICENSE-3RD-PARTY.txt — which is 404 now. Probably something's not built and/or deployed properly in our CI deployment process.

• Editor
  • ksy saved after modification
  • auto-clone files from read-only fs
  • tab renamed to ksy file name
• File system operations
  • files are visible (file list is not empty)
  • default kaitai.io samples and formats are there
  • new file, upload, download works
  • drag and drop works
    • if only one file was uploaded then it's automatically opened
  • right click context menu works
  • generating parser works
  • creating folder works
  • files are not overwritten on name conflict
  • context-specific icons (kaitai.io - cloud, formats - book, samples - cd, ksy - list-alt, non-ksy: newfile)
• Sample parsing works (zip)
• Converter panel works (every value is filled)
• Error panel works (open .ksy w/ other binary file) / it's closable
  • ksy parsing errors
  • file parsing errors
• Import works
  • relative import to a subfolder works
• About webide works / remember closing it for ever
• Hex viewer operations
  • mouse range selection works
  • deselect works
  • Two-way sync works between object tree and hex view
  • Right-click context menu works (selected data download option)
  • tab renamed to input file name
• Info panel works
  • selection shown, can modify
  • disable lazy parsing works
  • unparsed & byte array steppers work
  • selected path shown
  • export to JSON works
• Works in incognito mode (w/ empty storage)
• Proper warning without JS
• Works in supported browsers
  • Proper warning for unsupported browser
• Object tree
  • Shows partially parsed object tree on error
  • -webide-representation works
  • Can handle big files
    • Instances loaded lazily
    • Big arrays are split into smaller chunks (max 100 items in a level)
  • Two-way sync works between object tree and hex view
  • Open nodes are reopened after modifying source .ksy / reloading IDE
    • Even lazily loaded instances

Some formats depend on other .ksy format files.

The IDE should be modified in a way that we can load multiple ksy files for parsing even if we only use a single .ksy file for editing.

One of the possible approaches is to create a "loaded classes" window where we can control which classes are auto-loaded before the parsing happens. Here we should be able load, unload single classes and unload every currently loaded classes too.

Make compiler version configurable.

Stable should default to the latest stable compiler version, while devel should default the latest snapshot compiler version.

If the loaded .ksy's version is higher than the loaded compiler's version then suggest to the user to switch compiler versions.

Setup:
Windows 64 bit,
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, May 11 2017, 13:17:26) [MSC v.
1500 64 bit (AMD64)] on win32
node: v6.11.3
tried typescript: 2.5.2

I've followed the steps to compile web ide locally on Windows 7 but failed due to this error.
C:\Users\test\kaitai_struct_webide>python serve.py --compile auto compile started please use 127.0.0.1:8888 on Windows (using localhost makes 1sec delay) Changed file: ./src\worker\TemplateCompiler.ts 'tsc' is not recognized as an internal or external command, operable program or batch file. Compile errors: -

1. Obviously, the typescript compiler was not found but description does not mention there is a need to install typescript separately.
2. After installing typescript manually with
   C:\Users\test\kaitai_struct_webide>npm install -g typescript
   there were a bunch of weird errors during compillation process.
3. In addition, there is is a bug in genKaitaiFsFiles.py file since when executing command below from the steps there is an error.
   C:\Users\test\kaitai_struct_webide>python serve.py --compile auto compile started Traceback (most recent call last): File "serve.py", line 111, in <module> import genKaitaiFsFiles File "C:\Users\iscamc\kaitai_struct_webide\genKaitaiFsFiles.py", line 21, in < module> with open(outDir + 'js/kaitaiFsFiles.js', 'wt') as f: IOError: [Errno 2] No such file or directory: '--compile/js/kaitaiFsFiles.js'
   I was able to fix it changing in genKaitaiFsFiles.py
   outDir = sys.argv[1] + '/' if len(sys.argv) > 1 else ''
   to
   outDir = sys.argv[1] + '/' if len(sys.argv) > 2 else ''

Any thoughts how to resolve this?

Hi!

I'm loving the web IDE so far, but it seems imports aren't working.

If I copy the imports example from http://doc.kaitai.io/user_guide.html#_importing_types_from_other_files and open the filelist.ksy file in the IDE, the object tree is not updated (it stays blank or open on the previous ksy file) but there are no errors visible in the IDE.

If I open the browser console, I can see these errors:

As text:

localforage.js:836 [performance/29.891] YAML parsing took 1 milliseconds.
localforage.js:836 import yaml: date, mode: rel
localforage.js:836 import yaml: date, mode: rel
localforage.js:836 GET https://ide.kaitai.io/formats/date.ksy 404 ()
send @ jquery-3.1.1.min.js:4
ajax @ jquery-3.1.1.min.js:4
get @ /js/app.files.js:55
(anonymous) @ /js/app.js:37
importYaml @ /js/app.js:35
$c_Lio_kaitai_struct_format_JavaScriptClassSpecs.doImport__T__T__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:68200
$c_Lio_kaitai_struct_precompile_LoadImports.io$kaitai$struct$precompile$LoadImports$$loadImport__T__sci_List__s_Option__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7953
(anonymous) @ kaitai-struct-compiler-fastopt.js:7825
$c_Lio_kaitai_struct_precompile_LoadImports.processClass__Lio_kaitai_struct_format_ClassSpec__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7838
$c_Lio_kaitai_struct_Main$.importAndPrecompile__Lio_kaitai_struct_format_ClassSpecs__Lio_kaitai_struct_RuntimeConfig__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:4719
$c_Lio_kaitai_struct_MainJs$.compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__sjs_js_Promise @ kaitai-struct-compiler-fastopt.js:4892
$c_Lio_kaitai_struct_MainJs$.$$js$exported$meth$compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__O @ kaitai-struct-compiler-fastopt.js:4821
$c_Lio_kaitai_struct_MainJs$.compile @ kaitai-struct-compiler-fastopt.js:4945
compile @ /js/app.js:100
copyPromise.then.then @ /js/app.js:127
Promise resolved (async)
copyPromise.then @ /js/app.js:126
Promise resolved (async)
localforage.getItem.then.ksyFsItem @ /js/app.js:123
Promise resolved (async)
req.onsuccess @ localforage.js:836
IndexedDB (async)
(anonymous) @ localforage.js:826
localforage.js:836 XHR finished loading: GET "https://ide.kaitai.io/formats/date.ksy".
send @ jquery-3.1.1.min.js:4
ajax @ jquery-3.1.1.min.js:4
get @ /js/app.files.js:55
(anonymous) @ /js/app.js:37
importYaml @ /js/app.js:35
$c_Lio_kaitai_struct_format_JavaScriptClassSpecs.doImport__T__T__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:68200
$c_Lio_kaitai_struct_precompile_LoadImports.io$kaitai$struct$precompile$LoadImports$$loadImport__T__sci_List__s_Option__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7953
(anonymous) @ kaitai-struct-compiler-fastopt.js:7825
$c_Lio_kaitai_struct_precompile_LoadImports.processClass__Lio_kaitai_struct_format_ClassSpec__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7838
$c_Lio_kaitai_struct_Main$.importAndPrecompile__Lio_kaitai_struct_format_ClassSpecs__Lio_kaitai_struct_RuntimeConfig__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:4719
$c_Lio_kaitai_struct_MainJs$.compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__sjs_js_Promise @ kaitai-struct-compiler-fastopt.js:4892
$c_Lio_kaitai_struct_MainJs$.$$js$exported$meth$compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__O @ kaitai-struct-compiler-fastopt.js:4821
$c_Lio_kaitai_struct_MainJs$.compile @ kaitai-struct-compiler-fastopt.js:4945
compile @ /js/app.js:100
copyPromise.then.then @ /js/app.js:127
Promise resolved (async)
copyPromise.then @ /js/app.js:126
Promise resolved (async)
localforage.getItem.then.ksyFsItem @ /js/app.js:123
Promise resolved (async)
req.onsuccess @ localforage.js:836
IndexedDB (async)
(anonymous) @ localforage.js:826
/#:1 Uncaught (in promise) Object {readyState: 4, getResponseHeader: function, getAllResponseHeaders: function, setRequestHeader: function, overrideMimeType: function…}
Promise rejected (async)
j @ jquery-3.1.1.min.js:2
k @ jquery-3.1.1.min.js:2
setTimeout (async)
(anonymous) @ jquery-3.1.1.min.js:2
i @ jquery-3.1.1.min.js:2
fireWith @ jquery-3.1.1.min.js:2
fire @ jquery-3.1.1.min.js:2
i @ jquery-3.1.1.min.js:2
fireWith @ jquery-3.1.1.min.js:2
A @ jquery-3.1.1.min.js:4
(anonymous) @ jquery-3.1.1.min.js:4
XMLHttpRequest.send (async)
send @ jquery-3.1.1.min.js:4
ajax @ jquery-3.1.1.min.js:4
get @ /js/app.files.js:55
(anonymous) @ /js/app.js:37
importYaml @ /js/app.js:35
$c_Lio_kaitai_struct_format_JavaScriptClassSpecs.doImport__T__T__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:68200
$c_Lio_kaitai_struct_precompile_LoadImports.io$kaitai$struct$precompile$LoadImports$$loadImport__T__sci_List__s_Option__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7953
(anonymous) @ kaitai-struct-compiler-fastopt.js:7825
$c_Lio_kaitai_struct_precompile_LoadImports.processClass__Lio_kaitai_struct_format_ClassSpec__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7838
$c_Lio_kaitai_struct_Main$.importAndPrecompile__Lio_kaitai_struct_format_ClassSpecs__Lio_kaitai_struct_RuntimeConfig__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:4719
$c_Lio_kaitai_struct_MainJs$.compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__sjs_js_Promise @ kaitai-struct-compiler-fastopt.js:4892
$c_Lio_kaitai_struct_MainJs$.$$js$exported$meth$compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__O @ kaitai-struct-compiler-fastopt.js:4821
$c_Lio_kaitai_struct_MainJs$.compile @ kaitai-struct-compiler-fastopt.js:4945
compile @ /js/app.js:100
copyPromise.then.then @ /js/app.js:127
Promise resolved (async)
copyPromise.then @ /js/app.js:126
Promise resolved (async)
localforage.getItem.then.ksyFsItem @ /js/app.js:123
Promise resolved (async)
req.onsuccess @ localforage.js:836
IndexedDB (async)
(anonymous) @ localforage.js:826
localforage.js:836 GET https://ide.kaitai.io/formats/date.ksy 404 ()
send @ jquery-3.1.1.min.js:4
ajax @ jquery-3.1.1.min.js:4
get @ /js/app.files.js:55
(anonymous) @ /js/app.js:37
importYaml @ /js/app.js:35
$c_Lio_kaitai_struct_format_JavaScriptClassSpecs.doImport__T__T__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:68200
$c_Lio_kaitai_struct_precompile_LoadImports.io$kaitai$struct$precompile$LoadImports$$loadImport__T__sci_List__s_Option__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7953
(anonymous) @ kaitai-struct-compiler-fastopt.js:7825
$c_Lio_kaitai_struct_precompile_LoadImports.processClass__Lio_kaitai_struct_format_ClassSpec__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7838
$c_Lio_kaitai_struct_Main$.importAndPrecompile__Lio_kaitai_struct_format_ClassSpecs__Lio_kaitai_struct_RuntimeConfig__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:4719
$c_Lio_kaitai_struct_MainJs$.compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__sjs_js_Promise @ kaitai-struct-compiler-fastopt.js:4892
$c_Lio_kaitai_struct_MainJs$.$$js$exported$meth$compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__O @ kaitai-struct-compiler-fastopt.js:4821
$c_Lio_kaitai_struct_MainJs$.compile @ kaitai-struct-compiler-fastopt.js:4945
compile @ /js/app.js:101
copyPromise.then.then @ /js/app.js:127
Promise resolved (async)
copyPromise.then @ /js/app.js:126
Promise resolved (async)
localforage.getItem.then.ksyFsItem @ /js/app.js:123
Promise resolved (async)
req.onsuccess @ localforage.js:836
IndexedDB (async)
(anonymous) @ localforage.js:826
localforage.js:836 XHR finished loading: GET "https://ide.kaitai.io/formats/date.ksy".
send @ jquery-3.1.1.min.js:4
ajax @ jquery-3.1.1.min.js:4
get @ /js/app.files.js:55
(anonymous) @ /js/app.js:37
importYaml @ /js/app.js:35
$c_Lio_kaitai_struct_format_JavaScriptClassSpecs.doImport__T__T__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:68200
$c_Lio_kaitai_struct_precompile_LoadImports.io$kaitai$struct$precompile$LoadImports$$loadImport__T__sci_List__s_Option__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7953
(anonymous) @ kaitai-struct-compiler-fastopt.js:7825
$c_Lio_kaitai_struct_precompile_LoadImports.processClass__Lio_kaitai_struct_format_ClassSpec__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:7838
$c_Lio_kaitai_struct_Main$.importAndPrecompile__Lio_kaitai_struct_format_ClassSpecs__Lio_kaitai_struct_RuntimeConfig__s_concurrent_Future @ kaitai-struct-compiler-fastopt.js:4719
$c_Lio_kaitai_struct_MainJs$.compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__sjs_js_Promise @ kaitai-struct-compiler-fastopt.js:4892
$c_Lio_kaitai_struct_MainJs$.$$js$exported$meth$compile__T__sjs_js_Object__Lio_kaitai_struct_JavaScriptImporter__Z__O @ kaitai-struct-compiler-fastopt.js:4821
$c_Lio_kaitai_struct_MainJs$.compile @ kaitai-struct-compiler-fastopt.js:4945
compile @ /js/app.js:101
copyPromise.then.then @ /js/app.js:127
Promise resolved (async)
copyPromise.then @ /js/app.js:126
Promise resolved (async)
localforage.getItem.then.ksyFsItem @ /js/app.js:123
Promise resolved (async)
req.onsuccess @ localforage.js:836
IndexedDB (async)
(anonymous) @ localforage.js:826
/#:1 Uncaught (in promise) Object {readyState: 4, getResponseHeader: function, getAllResponseHeaders: function, setRequestHeader: function, overrideMimeType: function…}

It looks like it's trying to retrieve the date.ksy file from https://ide.kaitai.io/formats/date.ksy, but that's returning a 404 which is not caught.

Kaitai WebIDE version: 0.1.0.2207 (license)
Kaitai compiler version: 0.7 (2017-04-26 20:06:21.086)
Chrome version 58.0.3029.110 (64-bit)
Windows 10 64-bit

Because it is actually stored in the file system with the original name... And the FsItem should be used instead of the fn...

I'm looking at the demo site install at https://ide.kaitai.io/ and I attempted to upload a JPEG file from disk. The errors pane reports:
KS compilation error: io.kaitai.struct.format.YAMLParseException: /types/exif_in_jpeg/seq/1: unable to find type 'exif', searching from jpeg
The JPEG is well formed and has exif data.

Add option to parse

• part of the input file OR
• a hex-defined user input

as a defined type separately.

This can help quick prototyping and limit the possible external errors while parsing a complex file format.

I wanted to install webide on the android tablet using termux (I have tested webide remote version and find it quite useful, it looked to me to be less complicated to install with termux than cmd line version, as termux have no jde available). However I run into problem with installation and launching python scripts, which are written for python2, and only python3 comes with termux. Python3 is becoming a standard version in the many linux distributions and many software is dropping python 2 support.

So I can offer my time to modify python code and add PR, the main question is how? These are to considerations:

1. drop python 2 support, and convert python code to comply with python3 using e.g. py2to3.
2. if python2 support is wished to stay present, the python codebase could be modified to use six so that it would work with both python2 and python3.

PS. I am really impressed with capabilities of kaitai_struct (even if used only as the tool to RE binary format) it is manyfold better tool compared with other hex editors. I have some highly dynamic binary formats in my RE experience (some scientific electron microscopy formats, including virtual file system) and using Hexinator or wxHexEditor were really painful on dynamic data structs. Lately I am RE another binary format and find that using traditional hexeditors I am not able to manage whole complexity, and started to translate my knowledge about format into ksy. I wish kaitai_struct were available for me two years ago, I would had not lost so much time. I am still not sure about compiled code, as excessive class construction and hierarchical instance access is costly in python. Soon I will have something to compare.

Distribute the kaitai_struct_webide as Electron application and npm module. it should allow us install the kaitai_struct_ide as npm package, something like that: npm install kaitai (alternative names: npm install kaitai_struct or npm install kaitai_struct_ide).

Here is an example of wrapping it with electron:

#!/usr/bin/env bash

set -euo pipefail

git clone --depth 1 https://github.com/kaitai-io/kaitai_struct_webide.git
cd kaitai_struct_webide
cat <<'OUT' > package.json
{
  "name": "kaitai_struct_webide",
  "description": "Web IDE for Kaitai Struct",
  "directories": {
    "doc": "docs"
  },
  "dependencies": {
    "typescript": "^2.2.1",
    "electron": "latest"
  },
  "main": "main.js",
  "devDependencies": {},
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start-server": "python2 serve.py &",
    "start": "npm run-script start-server && node_modules/.bin/electron ."
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/kaitai-io/kaitai_struct_webide.git"
  },
  "author": "",
  "license": "GPL-3.0",
  "bugs": {
    "url": "https://github.com/kaitai-io/kaitai_struct_webide/issues"
  },
  "homepage": "https://github.com/kaitai-io/kaitai_struct_webide#readme"
}
OUT
npm install

Create slightly changed kaitai_struct_webide/main.js from electron quickstart:

const electron = require('electron')
// Module to control application life.
const app = electron.app
// Module to create native browser window.
const BrowserWindow = electron.BrowserWindow

const path = require('path')
const url = require('url')

// Keep a global reference of the window object, if you don't, the window will
// be closed automatically when the JavaScript object is garbage collected.
let mainWindow

function createWindow () {
  // Create the browser window.
  mainWindow = new BrowserWindow({width: 800, height: 600})

  // and load the index.html of the app.
  mainWindow.loadURL('http://localhost:8000');

  // Open the DevTools.
  // mainWindow.webContents.openDevTools()

  // Emitted when the window is closed.
  mainWindow.on('closed', function () {
    // Dereference the window object, usually you would store windows
    // in an array if your app supports multi windows, this is the time
    // when you should delete the corresponding element.
    mainWindow = null
  })
}

// This method will be called when Electron has finished
// initialization and is ready to create browser windows.
// Some APIs can only be used after this event occurs.
app.on('ready', createWindow)

// Quit when all windows are closed.
app.on('window-all-closed', function () {
  // On OS X it is common for applications and their menu bar
  // to stay active until the user quits explicitly with Cmd + Q
  if (process.platform !== 'darwin') {
    app.quit()
  }
})

app.on('activate', function () {
  // On OS X it's common to re-create a window in the app when the
  // dock icon is clicked and there are no other windows open.
  if (mainWindow === null) {
    createWindow()
  }
})

// In this file you can include the rest of your app's specific main process
// code. You can also put them in separate files and require them here.

Then run with npm start.

It should work but it shows the Your browser is not supported. Features may or may not working. Please use the latest Chrome or Firefox message. It is shown in other browsers too (e.g. Firefox 52), so I assume we need to check compatibility.

Current color scheme of web IDE is suboptimal: some of the widgets are clearly in "dark" theme (i.e. bright foreground on dark background), and some of them are in "bright" theme (i.e. the opposite). It's kind of hard for the eyes, as you can't get used to either of them.

I want to suggest that we could:

• develop 2 themes: one "all dark" and one "all bright"
• introduce a way to switch between them

When one's working on a moderately large project, it is often impractical to view the whole tree of resulting objects, when you're working on something deep inside it.

It would be cool if tree view window would allow to choose expression to visualize. I.e. by default it is something like obj (and it recursively allows to travel through whole tree), but I can choose it show the tree starting from

obj.foo.bar.baz[3]

or, even more complex example, something like

obj.records.filter(r => r.header.type == 42).map(r => r.body.data)

It should be relatively easy to do, if we'll just traverse object resulting from this expression, and not default object?

This way the WebIDE could save / load .ksy and input binary files directly from filesystem, so we don't have to upload / download files manually.

(If the local folder is synced then you can get version history, push directly to Git, etc.)

If we have 16 bytes that are supposed to be GUID/UUID, I propose to add some hint that would allow us to display it in its canonical text form, i.e. stuff like 123e4567-e89b-12d3-a456-426655440000.

Currently we need hacks like that to get some sort of approximation:

types:
  uuid:
    seq:
      - id: p1
        type: u4
      - id: p2
        type: u2
      - id: p3
        type: u2
      - id: p4
        type: u2be
      - id: p5
        size: 6
    -webide-representation: "{p1}-{p2}-{p3}-{p4}-{p5}"

However, it's better if we could get rid of special UUID type just for display purposes and get WebIDE to represent any 16-byte raw byte array as UUID using something like

-webide-representation: "{some_attribute:uuid}"

or something like that.

Load and save files (commit?) directly from a Github repo.

When reversing a format you often look at hex, trying to guess size based on visual perception, and look for that size (trying any endianness and any integer size in worst case) trying to locate the field which it stores. But there is no such a feature in webide.

Larger values are displayed incorrectly in the object tree pane. The converter pane shows them correctly.

The screenshot shows a selection of the bytes 0xbbbbbbbbaaaaaaaa as 0xbbbbbbbbaaaaa800 in the object tree.

require.js times out after ~30 seconds. Tries to reload every resource on refresh and times out again and again => no trivial workaround is known to solve this.

Although it looks like the new file is selected, when you modify the new file actually the old file is modified.

The converter panel counts bits (u32le) whereas KSY uses bytes (u4le). It also has separate columns for signed and unsigned values but labels rows with the signed type only.

Maybe add iconv-lite support too?

Make it possible and document an easy way to stop on exceptions (but skip compiler exceptions).

Hi all, I'm not sure it's an issue or expected behaviour but the fact is that I didn't have this issue in the web ide or vsc so I wanted to share with you:

I have this kind os sequence in my ksy:

- id : value1
  type : u1
  if : header.value1bit == true
- id : value2
  type : u4
  if : value1 == 0 

when evaluating value1 == 0 the csharp code wont check that it exists in the curent parsing context (only if header.value1bit == true) and would then evaluate value1==0 as true and the parsing is crashing.

I had to change it to:

  - id : value2
     type : u4
     if :  header.value1bit == true and value1 == 0 

I have the feeling that as we are referencing "value1" in the second expression, it should implicitly check that "value1" exists, kind of inherit that from the first expression...

Opening https://ide.kaitai.io/ in Firefox Nightly on Windows 7 64 bit, selecting formats/archive/cpio_old_le.ksy and I see this error and empty object tree

Parse error: undefined
Call stack: undefined io.kaitai.struct.format.YAMLParseException: /types/file/seq/6/if: can't compare BytesLimitType(BinOp(Attribute(Name(identifier(header)),identifier(path_name_size)),Sub,IntNum(1)),None,false,None,None) and CalcBytesType

https://gist.github.com/DarkDimius/9c8e32c9912d294600aa1d875cdb52d1 - ksy
https://yadi.sk/d/JkoqyX2A3ENycN - binary
https://yadi.sk/i/DZC3t9yc3EP6Xg - screenshot

quoting @koczkatamas

yeah, it's a bug (or a missing feature), what happens here is that the 'decoded' property is an instance, so I don't know the type of it only after processing (expanding) it, but after doing that I don't modify the node anymore, only add child nodes

We need feature usage metrics to make wiser development decisions.

We'll probably use Google Analytics for this purpose and it would be good if we could warn the user about this, explain why we use it and give opt-out option (other than blocking it by browser extension).

Make the WebIDE able to handle cases where custom processing is used like in this example: kaitai-io/kaitai_struct#137 (comment)

Try https://github.com/kaitai-io/kaitai_struct_formats/blob/master/scientific/nt_mdt/nt_mdt.ksy with http://www.ntmdt-si.ru/data/media/files/scangallery/graphene_and_carbon_materials/graphene_2.mdt

On 2 GiB RAM system it causes browser and GUI going unresponsive because of swapping.
Even if it is the result of a loop, I guess we need something to prevent this. For example webide can have some fields to limit amount of ram and stop parsing when the limit is hit.

P.S. python-compiled version works OK on that file.

TODO:

• Show doc contents when navigating the tree (in status bar / description field / something like that)
• Github integration for storing formats (w/ commit interface)
• Smooth experience with big files too (progress bar / cancelable operations)
• Breakpoint support for YAML window (injects JS "debugger;" into parsing code)
• Advanced breakpoint support: calls custom JS code (stored in separate file) instead of "debugger;"
• Save generated code (debug too): with filename ex. BmpDebugParser.py (where Bmp, Debug and py comes from the current settings -- somehow add other languages on GUI)
• Show parsing / compilation errors in YAML editor
• TTF: interval handling does not work correctly
• Show array count
• Fix negative enum handling

Not planned in the near future:

• [HexViewer] Search feature (hexbytes, ascii, utf8, utf16, integer, etc?)
• [HexViewer] Hex editor features - see also: #71
• Split hex view for data after conversion (base64, zlib, etc)
• Desktop clients for handling large files

Not planned (until explicit user request):

• Save layout, opened files, open tree nodes, etc (bonus: sync this to your account in the cloud and/or create a downloadable 'project file' from it)
  • opened files (ksy & input binary) and tree nodes are saved
• Tabbed view for multiple simultaneously opened files
• "w" and dump its contents into the file
• Enter when your cursor is on byte array field, and you'll get a full-screen hex viewer with its contents
• Multi-level struct coloring in hex viewer

Done:

• Change .ksy editor tab title to current format's filename
• Change input binary tab title to current input binary's filename
• Tree node: group large number of array nodes into chunks (0..99, 100..199, etc - like in Chrome console)
• Drag & drop format & input upload
• Remember latest format & input binary
• List sample formats and sample files
• Generate code for other languages
• Local storage based file storage
• Save read-only files after modification as a copy to Local Storage
• Support for adding new .ksy file on GUI
• Select tree node after hex view selection
• File upload w/ file browser
• File download from tree
• Buttons for file tree: new .ksy / upload / download
• Make intervals work
• Data view support for JS KaitaiStream (non-zero offset, limited length)
• Fix error window: parsing error is only shown in browser console
• Try to restore tree view after reparse (open same nodes)
• [HexViewer] Save selected data range
• [HexViewer] Jump to address feature
• No Javascript warning
• Unsupported browser warning
• [HexViewer] Show selected data in different formats (int16/32/64 le/be, float/double, strings: ascii, utf8, utf16, etc)
• Keyboard support for HexViewer
• Resolve enum names

Currently when opaque types/processes are used the script fails because it does not have definitions for opaque items. It would be nice if .js files could be loaded into the Web IDE and used to "fill in" opaque types and processes during KSY development. The Web IDE could scan the JS Code (debug) for opaque items by looking for "require('./SOME_FILE.js')" to extract the file name then identify if this external item has a KSY file. If it does then do nothing. If it does not then scan for a corresponding .js file and if found then copy js file contents to bottom of JS Code (Debug). The re-parsing would then work and allow users to test opaque items.

Currently, hex viewer always shows 16 bytes (0x10 bytes) per line. On bigger screens, it might be more comfortable to dump more bytes per line (i.e. 20, 24, 32 or even more). Can we try to make it a configurable setting?

http://requirejs.org/docs/api.html#config-waitSeconds

The project uses a lot of 3rd party libraries now. Probably we'll need some clear "licensing" text, that will contain all of these licenses with references to particular files, copyright holders, etc, etc.

js runtime relies on nodejs zlib +require module, neither are available in the browser.

Proposed fix:

• #46
• kaitai-io/kaitai_struct_javascript_runtime#4

Is there any open-source License for this project? There should be a LICENSE file in the root.

It is damn slow when I enter a char and it tries to compile an invalid ksy and parse a large file with it. It's better to compile and parse by ctrl+enter.

Create built-in ksy syntax analyzer based on the Ksy Style Guide (kaitai-io/kaitai_struct#140).

This is valid YAML/KSY that contains a flow-folded string^[1][2][3].

meta:
  id: example
  file-extension: example
seq:
  - id: stringdemo
    size: 0
    doc: This is a flow-style
      multi-line string

But causes the web IDE's YAML parser to throw a ParseException:
YAML parsing error: <ParseException> Unable to parse. (line 8: ' multi-line string')

Quoting the string (with ' or ") causes the parser to throw this exception instead:
YAML parsing error: <ParseException> Malformed inline YAML string ('This is a flow-style). (line 7: 'doc: 'This is a flow-style ')

1. Flow folding from official spec
2. Flow-style strings from official spec
3. Some friendlier unofficial YAML docs (The list in the second half of the 'Scalars' section in particular)