The tambo library is ready for a first round of code review. The library itself should be reviewed as common code, i.e. code that will be used in all simulations that produce sound, which at some point will likely be all HTML5 sims.

The demo code will not be published as a separate sim, and should be reviewed for its ability to serve as a good example of the usage of the library as well as a decent set of test cases.

The library is currently being used in the John Travoltage simulation, the reviewer is encouraged to look at the usages there and comment on anything that looks problematic or confusing.

There are some known features that will be added over time, and these will be reviewed as added.

As of this writing, the sonification manager (soon to be renamed sound manager) has no awareness of which screens are associated with which sounds. One of the early incarnations did have this awareness so that continuous sounds associated with a particular screen could be muted when that screen was not visible, but that was replaced by associating a sound with a Scenery node and disabling that sound when the node was not displayed in the scene graph.

The problem with this is that reset happens per screen, so if a sound needs to be muted during a reset, it should be disabled only when the reset occurs for the screen it is on. At the moment, this isn't how it works. Instead, the sonification manager amalgamates all the reset events for the screens into a single property that basically says, "a reset is occurring somewhere in the simulation" and passes it to all sound generators. This is awkward, and we should do it differently.

Several ideas were kicked around during the 6/14/2018 design meeting, and we decided to create this issue and track them, and then make a decision on the best approach.

In phetsims/vibe#28, we decided to add a query parameter that would turn sound all the way down regardless of what the user did. This was originally motivated by the desire to keep sounds from startling the bejesus out of us when doing automated fuzz testing, but we made the query parameter public and may have let outside users know about it.

We'll need this functionality - or some variation of it - in the tambo library. I personally would prefer just a flag that would completely remove support for sound, including preventing the icon from showing up on the nav bar, instead of a value that would turn the sound down. The latter seems like it could be confusing to users.

You may be inclined to ask why we can't use the current query parameter that enables sound. The reason is that this query parameter is intended to be override-able by the sim configuration, so it couldn't be used to disable the sound in a fully sound-enabled simulation.

My proposal is to add a query parameter flag called preventSound that, if present, would prevent the sound icon from appearing on the nav bar, would prevent the "Enhanced Sound" option from appearing in the PhET menu, and would prevent any sound from being generated.

Assigning to @emily-phet and @ariel-phet for input. We can make this a design meeting topic at some point if needed.

We recently decided that, due to lack of support for Web Audio, sonification won't be supported at all on IE11, see #6. However, sonification shouldn't prevent the sim from running at all, and I just tested the current master version of the demo (which is probably about the same as v0.0.0-dev.0, but I can't test that because bayes is down), and it doesn't load. I suspect that I'll need to add some sort of a faked-out audio context to work on IE11.

@Ashton-Morris recently asked me whether we would be able to use sound effects from a set that was made available by the BBC, which can be seen at http://bbcsfx.acropolis.org.uk/. The license is "RemArc" and is described here: https://github.com/bbcarchdev/Remarc/blob/master/doc/2016.09.27_RemArc_Content%20licence_Terms%20of%20Use_final.pdf.

Is this license compatible with PhET, and could we use these sounds in our simulations?

We are using a number of different sound effects in various sims, and the number is growing as we experiment with sonification. The loudness of the sound effects varies quite a bit, which makes them challenging to mix in a sim, so I think it would be good for us to standardize on some value. I took a quick look around on line and found a short article on the topic, see http://www.creativefieldrecording.com/2017/05/10/2-techniques-for-setting-sound-effects-library-mastering-levels/.

I'd like to get input from @Ashton_Morris and @mikewinters10 to see if they have any insights.

At the review meeting on 7/5/2018 we decided that the sound on/off icon should be changed a bit to look more compatible with the keyboard icon, which it will often be beside on the nav bar. Specifically, it should have thinner waves and a thinner X that look more like the stroke used on the keyboard icon.

To duplicate, go to the 2nd screen of https://phet-dev.colorado.edu/html/tambo/0.0.0-dev.1/phet/tambo_en_phet.html and switch the A/B switch back and forth. Much of the time when the sound is turned off a sort of click can be heard. I think this is because calling AudioSoundBuffer.stop causes an immediate stop, which probably sets the output immediately to zero. We should probably use a combination of a fast but not instant reduction in output gain level and a scheduled stop in order to make this sound better.

This was discussed in the 7/5/2018 sonification design meeting, and there were a lot of ideas, and eventually we decided to log an issue and assign it to @arouinfar for a mockup or two. Here is a screenshot from the master version of tambo:

In general, folks are thinking that the icons should be vertically aligned with the letters in the PhET logo.

Chrome is adding a feature to block autoplay of audio when a website loads, and that unfortunately prevents sound generation from our simulations. Code has been added to the published sound generation code in the vibe repo that turns sound on once the user interacts with the sim, see phetsims/vibe#32. That same fix needs to be incorporated into the sonification manager.

I built a phet-brand version of John Travoltage, then used js-beautify to give helpful line numbers like so:

npm install js-beautify
node node_modules/js-beautify/js/bin/html-beautify.js john-travoltage/build/phet/john-travoltage_en_phet.html > john_travoltage.html

Then I tethered an iPad2 running iOS 9.3.5 to Safari 11.1.2 running on Mac OS 10.13.6. The sim launched and ran properly, and I am able to use the arm and reset all button. However, when dragging the leg, an error is produced. I noticed these messages in the console:

The highlighted line is:

this.isPlaying || (this.noiseSource = this.audioContext.createBufferSource(), this.noiseSource.buffer = this.noiseBuffer, this.noiseSource.loop = !0, this.noiseSource.connect(this.noiseSourceConnectionPoint), this.noiseSource.start(t), this.isPlaying = !0)

The failed addSoundGenerator line is:

                            addSoundGenerator: function(t, e) {
                                if (!b) return console.warn("an attempt was made to add a sound generator to an uninitialized sound manager, ignoring (sound will not be produced)"), null;

I just incorporated a new impulse response into the sonification manager. I got this impulse response from a site that says that the file is CC Public Domain. I'd like to double check that we can indeed use this recording and that the license entry is correct. @samreid - since you seem to know the most about licensing, please take a look at tambo/audio/license.json, check out the entry for empty-apartment-bedroom-06.mp3, and let me know if all is good. There is a link to the location where the sound was obtained in the license file, but in case you need it here, it came from http://www.openairlib.net/auralizationdb/content/empty-apartment-bedroom.

This was reported during multi-platform testing of a dev version of the tambo demo, see phetsims/qa#160 (comment).

Loop sounds that sound good on one platform often have a gap or a click when tested on another (e.g. Chrome vs Firefox on my Win 10 machine). There is a lot of talk on line about how different mp3 decoders insert different amounts of silence when encoding, and I suspect that there is a similar effect from different decoders.

As of this writing, there are two types that support the playing of pre-recorded sounds, one for one-shot sounds and one for loops. These share a common base class, so much of their code is already common. In the sonification framework review meeting on 9/6/2018, the development team expressed a preference for having these be a single type with options. I had split them out a while back because of differences in the way I thought the API should behave, but in the discussions at the meeting we worked out how to handle the differences, so I'm moving forward with this.

@jbphet
I'm writing the annual report for the grant that funds the sonification work. I'd like to include a section that describes Tambo. Could you draft up a 1-page document that includes a:

• Brief description of the relationship between Tambo and phet’s overall infrastructure
• Brief description of how Tambo is structured
• Summary or list of affordances of Tambo (for example, the relevant features it supports that we found necessary from the sonification work as a whole).

Audience:
Option 1) Try to target this towards a lay audience, though it's ok if some parts won't be completely understandable to everyone. The idea is to convey an idea of what was created under this grant, not that the reader should be able to utilize this information directly in their work.

Option 2) Alternately, you might consider if this type of information (written with developers in mind) would be useful for some supporting documentation on GitHub. If that's the case, could can write a version for GitHub, and I can adapt (reword/simplify) for the grant report.

If you could get your text to me by August 29th, that should leave me enough time to adapt or revise the text before including it in the report (to be submitted Sept. 4th).

Of course, let me know if you have any questions, concerns!

The current implementation of SoundClip decodes the sound audio data at construction in a Web Audio AudioBuffer, and each time the sound clip is played a new AudioBufferSourceNode is created to play the sound. This is done so that the sound can be played close together without canceling the playback of the existing sound. This is how most Web Audio examples for playing pre-recorded audio work.

At a recent review meeting, the ability to stop individual clips was requested. The way we decided to support this was to return an object when play() is called that the client can use later to stop the sound.

At the time of this writing, none of the existing sound designs need this feature, but it may be needed in the future.

Test device:
Schrodinger and Bates
Operating System:
Windows 8.1 and 7
Browser:
IE 11
Problem description:
When trying to take away balls in the first tambo screen the sim crashes. For phetsims/qa/issues/160
Steps to reproduce:

1. Go to the first screen. Have the console open.
2. Add several balls to the box
3. Try to take one away

Screenshot:

Video

There is a JavaScript library called "Flocking" that is based off of an open source product for sound generation called "SuperCollider". It may work well for us to use this library, since some of the people with whom we are collaborating may be using SuperCollider to design sounds, so using flocking may give us a reasonably easy path for getting the designed sounds into an actual sim.

There are several major issues that would need to be resolved to decide if we really want to use Flocking, but this issue is concerned solely about whether the licensing supports its usage in PhET sims.

Link to GitHub repo: https://github.com/colinbdclark/Flocking

Sound on/off and sonification level (basic or enhanced) are global parameters that are being added for sonification. These should be initially settable via query parameters.

Related to code review #39. This will be a list of ES6 conversions to consider. I'll be editing this list while I'm doing the code review.

• define( function( require ) {...} to define( require => {...} )
• inherit to class
• return tambo.register( 'foo', foo )
• get rid of self where not needed
• get rid of function where you can

The trimSilence option, as of this writing, removes silence from the beginning and end of loops. I should make it so that it also removes silence from the beginning of one-shot sounds so that they play more quickly.

Related to code review #39.

Boolean names are a bit inconsistent. E.g. SoundClip has isPlaying, while SoundGenerator has locallyEnabledProperty and fullyEnabledProperty. Should all boolean names have an "is" prefix? That was the message that I got when I worked on phetsims/scenery-phet#388 recently. On the other hand, we've consistently used enabledProperty rather than isEnabledProperty.

This might be a good issue to get input from other developers.

In the 8/30/2018 sonification design review meeting we collectively decided on an approach to handling some query parameter support for the sound feature. The current plan is to have a 'sound' parameter that can be set to 'enabled' (the default), 'disabled', or 'muted'. Enabled would be the case where sound is initially enabled. The 'disabled' value would mean that sound is not available at all, and the icon to control is would not be present on the nav bar. The 'muted' value would mean that sound is available in the sim, but is initially muted and must be un-muted by the user by clicking the nav bar control.

The supportsSound and supportsEnhancedSound query parameters were added a few months back as a way to support the development of sonification on the master branch for a simulation without enabling sound generation by default. The idea was borrowed from how accessibility is currently being developed. The primary motivation for this is to prevent the accidental deployment of a sim with an unfinished sonification implementation.

In a recent discussion, @samreid suggested that we get rid of these query parameters and control sound behavior entirely through options to the Sim constructor.

While @samreid and I were working on the integration of the sound controls into joist, he suggested that we change the query parameter and configuration flag to enabled sound support from "tambo", which reflects the library name, to "supportsSound". I quite like this. The downside is that it could potentially be confusing in pre-tambo sims with sound support, since they would still support sound even without this query parameter or configuration option.

I'd like to discuss in the next dev meeting and, unless there are strong opinions against the use of "supportsSound", I'd like to go with that. I think developers can handle the potential confusion, and the code with this flag will be easier to understand in the long run.

I would like to discuss possibilities for allowing our collaborators on the sonifcation project to use iO features to accomplish some sonification.

This strategy seems to have a couple of merits:

1. The researchers (I think) could have direct access to the data output of the sim and sonify that data externally, without us having to bake sonification into the sim. This approach would allow for rapid prototyping (and ideally relieve us of undo developer burden)
2. It would allow researchers to use 3rd party tools to create the sounds/soundscapes
3. Other people would be "shaking down" the API which seems beneficial
4. It shows off some more of the possibilities of the iO features

So...

• Does this approach seem feasible?
• Are there licensing issues? (I assume we would need to provide our "research" license)
• Roadblocks/problems?

The title states the essential question. The tradeoffs are:

• Adding it to all sims will increase the size by some amount and possibly the load time, since the sound manager will be creating an audio context, some gain stages, and convolver, etc. This could be moved into the initialize call if desired.
• Adding it only to simulations that use it will cause Sim.js to be a bit more complicated, and the main file for the sim (e.g. build-an-atom-main.js) will have additional code as well to pass in the sonification manager.

I'll gather some data, then put this on the agenda for the next framework design meeting. For now, I'm going with the "add it only when it is being used" approach to minimize impact on existing sims.

The behavior of one-shot sound clips was discussed in the sonification framework design meeting on 6/14/2018, and here are some things that need to be done:

• Sound generators should have an option that determines whether sound generation can be initiated when the sound generator is enabled. This makes it so that if a sound that is long is initiated when sound is disabled, then sound is enabled, the "tail" of the sound will be heard. The default should be that sound generation can be initiated when the sound generator is disabled.
• Disabling a one-shot sound generator after it has started generating its should should terminate the sound.

Version https://phet-dev.colorado.edu/html/tambo/1.0.0-dev.1/phet/tambo_en_phet.html has a lightning button on the 2nd screen and a checkbox to control whether thunder occurs. It does not behave as described above, so it can serve as a reference point for before-and-after comparison.

As of this writing we have two global parameters for controlling sound, one that controls whether it is globally enabled or disabled, and one that controls whether the "basic" or "enhanced" level of sonification is provided. We should decide what the default values are for these parameters.

I think we've essentially agreed that the default value for the level should be "basic", so please speak up if you disagree with that, otherwise I'll go with that.

I'm less clear on how enabled/disabled should work by default. In our currently published sims that provide sound for games, the default is to have sound generation enabled. In at least one sim - Plinko Probability - sound generation is off by default. My thought is that we would probably have sound enabled by default, but the bouncing of the balls would be part of the "enhanced" level in sims like Plinko so that they don't produce too much sound in the default configuration.

@emily-phet and @ariel-phet - can I get your input on this?

Many MP3 encoders, including the LAME encoder that is often used in conjunction with Audacity (a popular open-source sound editing program) insert silences at the beginning of an MP3 file to allow the encoder to start up. Here is an discussion on this topic: https://superuser.com/questions/938782/audacity-adds-a-gap-when-exporting-mp3.

This behavior can be problematic when looping, since the inserted silence often causes gaps in the loop. I noticed this when working on testing different encoding rates (see #28), and found it to be particularly bad for lower quality mp3 files. If I added an option that allowed the initial portion of the data to be trimmed once loaded into a sound buffer, I think this problem would go away, or at least be much less obvious.

Internet Explorer does not support the Web Audio API. In the previous work that was done to support sound in our simulations, a fallback case was added to vibe/Sound.js to allow pre-recorded sounds to be played in this browser. As support for more elaborate sounds is integrated into PhET's code base, this workaround becomes harder to maintain, and may cause odd situations, since we intend to have pre-recorded sounds used in conjunction with synthesized sounds, and it will not be possible to create workarounds for the latter. Here are some possible approaches to this problem that I can think of, others can add more if they have ideas. These are listed in order of increasing development effort.

• drop support for IE entirely
• drop support for sound in IE, so users of this browser would only experience the visual aspects of the simulations
• add an option to the SoundClip constructor that would indicate whether it should be played in the fallback case, and identify the sounds for which this should be set during the sound design process
• continue to maintain support for pre-recorded sound clips, and play them as appropriate, but keep in mind that ONLY pre-recorded sounds would be supported and some situations with partial sonification will result

Assigning to @emily-phet, @ariel-phet, and @kathy-phet to get their input.

To support phetsims/vibe#33

@mattpen can you apply the standard label set to this repo?

Loops can be started multiple times on the current implementation, which leads to odd behavior such as that reported in phetsims/qa#160 (comment). There should be something that prevents a loop from being played if it is already playing.

In #6 we decided that we will stop supporting sound on IE when the sonification library is integrated into the mainline sims. We are in the process of doing this integration now, and in the 8/23/2018 dev meeting we decided that the sound control button shouldn't appear at all on IE so that users aren't confused.

Sonification supports two levels, "basic" and "enhanced". The user needs to be able to select the setting.

We have a contractor who is doing some work for the sonification effort, and he has a library (and maybe several) that he would like to draw from. How do we determine whether sounds from a given library can be used? As we figure this out, I'd like us to keep track of the specific library or libraries in question and log our determination in this GitHub issue.

We agreed in a recent dev meeting to indicate whether query parameters are internal or public facing. This was not done for the new tambo and supportsEnhancedSound query parameters.

I've been documenting like this:

    /**
     * Blah blah blah.
     * For internal use only.
     */
    someQueryParameter: ...

    /**
     * Blah blah blah.
     * Public facing.
     */
    anotherQueryParameter: ...

Related to code review #39.

TamboKeyboardHelpContent lives in tambo/js/demo/, but isn't enabled in the demo by default. You need to add the accessibility query parameter. Consider enabling it by default, by passing option accessibility to Sim in tambo-main.js, ala john-travoltage-main.js.

A while back it was observed that the size of the john-travoltage sim nearly doubled when prototype sonification was added (see details below). It may be that other libraries grew at the same time, but it is important for us to understand the cost of adding sound in terms of file size. We may be able to make some tradeoffs in terms of encodings, stereo versus mono, etc. in order to manage these costs well.

The size change was determined by looking at JT v1.2.6, which is that last version without the prototype sonification, to JT v1.3.0, which was the first version with it. Using the du -h command on phet-server, v1.2.6 is 1.2M and v1.3.0 is 2.4M.

Sonification enabled/disabled is going to be a global setting. How should the user control it?

In working with a sound designer contractor on finalizing sound designs for John Travoltage, we ran into some new information we hadn't discussed previously when deciding what sound assets to archive.

Essentially, the sound clips we will be using are more like 'music' (e.g., like hiring a musician to create a song, and this does not come with the rights to all materials used to make the song, or editing files associated with 'tweaking' the song), than they are like artwork created by PhET designers (e.g., artwork is created from scratch in Illustrator, and all original files are kept and available for 'tweaking' or using as a starting point for new artwork in the future).

More specifics here, if you're interested
{
The sound designer utilizes a certain digital audio workspace named Reaper. The way the contractor made many (all?) of the recorded sounds (presumably this does not apply to sounds synthesized directly in the sim), was to have a starting sound or sounds imported into Reaper. They then manipulated this sound or sounds, sometimes using professional (for fee) manipulations. Once the desired outcome was reached, a new file is generated, which is the final sound clip used by PhET. The input files, manipulations, and Reaper itself all have licenses consistent with our needs, but it is not the case that all source files are CC-BY, or should become CC-BY. As another example, one of the sounds in John Travoltage is essentially the outcome of manipulations of a self-recorded nature sound by the sound designer. The sound designer (reasonably) does not want the nature recording to be open source, or to give rights to us beyond use of the outcome sound file for JT.
}

We have decided for the time being to treat the outcome *.wav file as the asset (this is the actual sound clip used within the sim). I have also asked the sound designer to document for each sim sound what the source sounds are, and their origins (e.g., self-recorded, openly licensed database of sounds, etc.). I'll probably follow up and ask the contractor to list the specific licensing for each (e.g., declare a license type for his self-recorded files).

I would like to discuss this in greater detail with whoever is the appropriate person, @kathy-phet, @ariel-phet ? Anyone else?
As a related (smaller) topic, this also brought up some questions for me about what assets in general we're keeping public vs private (e.g., assets related to PhET branding, etc.). I'd like to discuss this as well, but again, I think this is a minor topic.

At a recent review meeting the ability to pause, rewind, and resume the sound from a sound clip was requested. This seems like a good idea. At the moment, none of the sound designs need this feature, but I'll implement it soon anyway so that it can be tested and reviewed before it's needed.

Related to code review #39.

initialize-globals.js contains query parameters soundInitiallyEnabled and enhancedSoundInitiallyEnabled, used exclusively in soundManager.js. I think it would be fine to drop the "Initially" part of these names. It would still be obvious what they are for, and it would be less verbose.

I'm about to add code to handle removal and disposal of sound generators, and I thought it would be good to have a record of how much memory the sound generators use to make sure that they don't consume an inordinate amount of heap memory in the browser.

I'll use a published version for this so that it can be repeated, so either we will need to publish a one-off (which we may do for looking at the nav bar controls anyway) or wait until the sound controls are merged into master on joist (see phetsims/joist#497).

This CT failure is occurring regularly in tambo and sporadically in various other repositories (graphing-slope-intercept, gravity-force-lab-basics, john-travoltage, wave-interferences, wilder, ...)

tambo : fuzz : require.js : run
Uncaught TypeError: Cannot read property 'autoDetectLoopBounds' of undefined
TypeError: Cannot read property 'autoDetectLoopBounds' of undefined
at AudioContext. (https://bayes.colorado.edu/continuous-testing/snapshot-1535928147520/tambo/js/sound-generators/SoundClip.js?bust=1535971538352:60:22)
Approximately 9/2/2018, 4:42:27 PM

Related to code review #39.

SoundClip.js contains some private "helper" functions that are related to finding where sound starts and ends, etc. The functions are findSoundStartIndex, findSoundEndIndex, detectLoopBounds, and (possibly) logLoopAnalysisInfo. Consider moving these functions to another file, something like SoundUtils.js or whatever name you feel accurately describes this collection of helper functions.

Factoring out these functions would reduce the size of SoundClip.js by 100+ lines.