I should switch to WebGL to render the notes more efficiently, and to make it easier to implement fancy effects.

I don't know if a framework like PIXI.js would be helpful here, probably I can do it in raw WebGL. Well, might as well use THREE.js, and have 3D possibilities, both for effects, and for controls for lining up with a camera's view of a piano. Well, I'm not sure what controls that would be, but at any rate I'm familiar with THREE.js and know how I could implement note drawing efficiently (filling data in a BufferAttribute, and extending the draw range of the BufferGeometry for new notes. Notes are basically lines, or if I want to draw them as an extruded square, I believe I can do that with a vertex shader, so I'm still passing only the note data to the GPU. Not sure that 4x would matter though.)

I've wanted this simply for the look, and also for screenshots, but now I especially want it for live streaming - using the MIDI visualization as part of an OBS scene, overlaid on top of a camera video feed of the keyboard.

So far I've been using this:

document.querySelector("canvas").style.zIndex = 1000;
document.querySelector("canvas").style.background = "black";

and for toggling it off:

document.querySelector("canvas").style.zIndex = "";
document.querySelector("canvas").style.background = "";

This could be a fullscreen button that fullscreens the canvas specifically.

It would also be nice if the UI faded out or minimized somewhat when you start playing. We want the Save MIDI to still be clearly visible I think, but other than that, most of that stuff is like an about screen.

Right now MIDI devices that are not working properly (no port open) are shown with a red dot, rather than green (for open port).
This means I'm essentially indicating that it's not recording from a device by showing a red circle. I suspect this choice of iconography may confuse new users. I haven't done any user studies, but it seems like a bit of an incongruity.

When addressing this, I should also consider the prospective feature of ignoring specific MIDI devices, as this may influence the design. (The toggle would be on the same row as the port connection indicator.)

Could be some cool skeuomorphic designs here with plugs connecting/disconnecting. Like a MIDI patchbay, maybe — but I've never used such a thing and wouldn't want to create other weird semantic mismatches relating to ports vs channels or whatever.

As long as this doesn't complicate the app too much, it'd be nice to be able to play back recordings, for these reasons:

• In case you don't have software installed to play MIDI files (such as VLC, configured with a downloaded soundfont)
• If it's easier to use than such software, due to the convenience of being built-in
• To use the visualization to render video after-the-fact
  • Most playback software doesn't support MIDI visualization, especially of pitch bends.
  • This would be an easier workflow for video production than recording live, as it's hard to get the timings all lined up without latency, especially if you want notes to fall down onto the piano, rather than going out from the piano, which requires a screen's worth of latency for the notes to fall down, or magical future-prediction powers.

What this might look like:
I could add a "MIDI file playback" heading and file selector to the Input section, but there would need to be controls for pausing/playing, stopping, seeking, zooming, and volume adjustment; and those aren't "input" in the same way; they aren't MIDI data sources.
So I might need a new "Playback" section, at which point it might make sense to have the file input there, I'm not sure.
There should also be a way to play back what you just recorded (without saving and loading a file). Naturally playback can work using the recording, until you load a file, but should it automatically revert to using the recording buffer if you play notes? My gut says no, since you could have a MIDI device sending a reset signal before automatically powering off, which shouldn't interrupt your listening; or you could want to play along with a recording, to make a second part. Maybe if playback is finished, though, it should reset.
And there would need to be a soundfont / playback engine. There are libraries for this part.

Alternatively I could make a separate app for playing MIDI files, reusing the MIDI visualization between apps. This wouldn't have the convenience advantage of built-in playback, unless there's some linking/communication between the apps.

I don't know exactly in what situation this occurred, maybe I powered off the keyboard while holding notes as a sharp casual end to playing - probably that should reproduce - anyways this is what happens currently, the notes stay held:

Related: should probably release held notes when exporting MIDI (at least, treat it as releasing the notes, i.e. include note-offs) - some DAWs will probably leave the notes held when the MIDI clip ends, which, whilst not incorrect, is probably not what you want.

I've noticed that Chrome likes to crash a lot when doing long recordings, but I'm not sure how much of that is due to memory usage.
It would be good to investigate this directly, using memory profiling, and test data.
I have a demo mode, accessed with demo() in the console, but I suspect this may be a bug in Chrome's MIDI handling rather than
a memory usage problem, since it's not just the tab that crashes, but the entire browser.
(That said it could be a combination of things somehow, and it would be cool if the app could use the chunk storage feature (currently only for backup) to create arbitrarily long recordings; the problem is in reconstructing them; it'd need a streaming download, which there are new APIs for, but the MIDI file library probably doesn't support.)

Currently, start/end times are determined from a timer separate from the MIDI clock; it uses performance.now() so it records the time the event was received, or rather handled. This is the only straightforward way to do it, because I don't think the Web MIDI API gives you timestamps!

But what happens is if the page lags, the notes build up and all appear to start/end once the lag is over. (and are recorded as such.)
This is not good!

Investigate:

• Is it possible to access WebMIDI from a WebWorker?
• Is there any better timer to use, like the clock signals in MIDI? Can we access that and use it to correct for lag?

With ChromeVox at least, the Apply button is not announced when you activate the Learn Range feature, because it uses the same button and just changes the text label.

I should either make it a new button, or perhaps unfocusing and refocusing it would work fine.

• Instrument change (displayed as a bar with instrument number and name)
• Instrument bank select
• Aftertouch pressure (I don't have a device that supports this)
• Investigate: can we more simply handle MIDI file creation, especially with all possible events, if we ditch SimpleMidiInput? How low level is the WebMIDI API? How well do MIDI streams map to Standard MIDI Files? Also, what about JZZ.js?

Chrome doesn't work with OBS due to hardware acceleration, so I want to use this in Firefox.

I can use Display Capture, but Window Capture would let me see the MIDI visualization in OBS while OBS is focused, without adding another monitor to my computer - and it'd be less prone to showing information I don't intend to, i.e. windows I have open while alt+tabbing.

Framegrab of me using the setup:

What OBS looks like if I switch to OBS:

Some pics of the setup for context:

It's held up with a mic holder that has a clamp end, shimmed with a scrap of cloth around the end of the PVC pole, and some bits of rubber to protect the shelf it's clamped to, and held up with a single piece of string. I happened to have a hook and a window curtain rod bulb in opportune places. The hook was way further away, but I just wrapped it around from the shorter side a second time and that balanced it out.

(Note: lighting is not good here; the open window there is not helpful, and the lamp only helps compliment the artificial (ceiling) light at night, filling in the shadows that the black keys cast.)

The Error

TypeError: elToReplaceContentOfOnError is null

erm, well that looks like an error handling error
presumably there's some error behind that...

Currently the workflow of the app is that it automatically records MIDI, but requires one click to save, and one more click to clear.
Historically there's been no way to reliably download files outside of a user gesture, an annoying but understandable restriction.

Using the FS Access API, I could add an option where you can specify a directory to save files, and then automatically write files to this folder when a configurable period of silence is detected, i.e. when no note is pressed for a given time.

As for the naming of files, it could be left to the user to rename files after they're created, but this will probably lead to a less organized record collection, at least for me personally, as I usually don't listen back to my recordings to figure out what they are.
(I could also implement some automatic naming based on detected key signatures and tempos, maybe use AI to categorize songs based on the names and associated content of previous recordings in the folder. But that's clearly an extra feature.)

I was never able to get the Firefox Web MIDI browser extension to work on Ubuntu, but it works on Windows.

I have some code that's supposed to refresh the page automatically when no MIDI devices are detected, but it's refreshing too often (and while TASCAM US-122L MIDI is present), although it did stop once I turned on my MIDI keyboard and started playing.

It should probably have a time limit at least, stored across page reloads.

Cool app! I tried this with e-drums and it works, but the note lengths are strangely variable. This isn't your fault, it's just what the instrument sends (Yamaha DTX). But it makes it so you can't see the rhythms, it just looks like a mess. My idea for drums is just to give each note a fixed length, ignoring the end points coming from the instrument. They should be fairly thin so you can see fast rhythms. Maybe configurable.

Right now split/dual settings (multi-voice features) don't work, everything goes into one channel; and multiple MIDI devices also go into one channel.

• When notes and other events are recorded from multiple MIDI devices, tracks should be created per instrument.
• When split/dual instrument settings are used on a piano, it should record into multiple tracks (or channels?)
• Color notes by channel / instrument in the visualizer

The names of tracks/channels could be based on the device name or instruments selected.
Instruments can change over time, but could be concatenated in the track name. Could be shortened if instruments fall in the same group (e.g. "Brass").
Device names are a little weird to include, almost a privacy issue ­— but probably fine, right?

"connected <device name>" kinda makes sense, so I left it like that for a while, but, it's really not supposed to be announcing the fact that you connected a device, but that a device is now connected/available.

Particularly, if you turn on a device, rather than plug it in, "<device name> connected" should make more sense. It's like, the device is connecting to the system, it's not you (or the application) explicitly connecting the device.

In the app, I recommend using the Jazz-Soft browser extensions, but I haven't actually had any success with them.
The best I can get is the devices showing up in the list, but they show as disconnected.
I may be running into jazz-soft/web-midi#4

Hi Isaiah @1j01!

Thanks for building this awesome little midi-recorder. I've been using it for a while to record my piano improvs. Recently, however, it's started throwing an error on load. FYI, I'm using a Behringer
UMC404HD 192k audio/midi interface with Chrome Version 81.0.4044.138 (Official Build) (64-bit) on OSX High Sierra (10.13.6).

The error is:

Uncaught TypeError: Cannot read property 'value' of undefined
    at update_options_from_inputs (http://localhost:8000/app.coffee:260:2)
    at eval (http://localhost:8000/app.coffee:289:1)

The last known good version which worked for me was commit: c6f49b48341d29dafc0a431b7e953d40bfb07c2a

I've done a teeny bit of digging and it looks like various options aren't getting set around here:

If I comment out this line it loads, identifies my midi controller, and records input (but obviously doesn't update any config options):

Any ideas what might be going wrong?

Thanks again!
Peter

I try to calculate the duration that the MIDI file should be, included as a delta of an event of type MIDIEvents.EVENT_META_END_OF_TRACK in the "conductor track".
I could try and fix this and figure out what's wrong with the math, but I might be able to just get rid of this entirely, I'm not sure it's needed.
I also might be able to calculate this more easily by just totalling up the deltas of the events...
Looks like it should just need / ms_per_tick tho?
(assuming rounding errors don't come into play at these time scales?)

It would be nice to be able to scroll back through what you played,

It should automatically pause the visualization, stop it from scrolling through time, but it shouldn't pause the recording process.
There should be an easy way to go back to the end, i.e. now, and resume auto-scrolling.

Questions:

• The UI can also scroll, if it doesn't all fit on screen, so how should you indicate to scroll the visualization versus the UI?
  Also you may expect scroll wheel to zoom the visualization, changing the time scale instead of the time offset.
• Discoverability: How to indicate that this feature exists?

Related: #16

This probably has to be a manual setting, but it would be nice if we can also detect if you change the setting on your device; there may be a MIDI event for this.

This is not very important since +/- 2 semitones is I think quite common and useful.

with a nice "Ready to use offline message" like in svgomg etc.

of course service workers can be a can of worms, a bit of a footgun, a minefield,
but it would be nice!
since it's an application that doesn't even use the network at all
and it's something I want to have ready at a moment's notice (inspiration strikes!), so I don't want network connectivity to be a concern when pulling it up

Would make it easier to use the MIDI visualization in OBS - rather than type pixel values until you find a good crop range, you could press keys on your keyboard to set the range.
And it would also be good for standalone usage, letting you make the MIDI viz as big as possible.

Should store the values in URL params so you can bookmark it with the right values.
...And/or localStorage?

Oh also we could automatically set the range based on known MIDI keyboard models...