Description

Using the existing prototype and the various layout demos, agree and prototype the basic app layout.

Resources

Full set of interesting links:

https://deploy-preview-7--eager-goldberg-513fc4.netlify.app/
https://deploy-preview-3--eager-goldberg-513fc4.netlify.app/
https://deploy-preview-4--eager-goldberg-513fc4.netlify.app/
https://deploy-preview-5--eager-goldberg-513fc4.netlify.app/
https://deploy-preview-6--eager-goldberg-513fc4.netlify.app/

https://eager-goldberg-513fc4.netlify.app/ is a moving target evolving as we get consensus.

See also #11

Todo

Iterative discussion. When we have a clear direction we can plan some new work.

Live issues:

• Tablet layout. See #11, #22.
• Zoom buttons via serial.
• ... ?

Conclusion

Description

Document editor component requirements and review the major contenders against these requirements.

Todo

• Document requirements
  • https://paper.dropbox.com/doc/Editor-component-review--BH6cVak976UvUcizMnCi7lskAg-xGNMaM8jV89tFUwWfmrpQ (WIP)
• Ace
• CM6
• Monaco

Conclusion

Bug Description

The editor options are only working because we're recreating CodeMirror. If we fix it not to do that (as that has other unfortunate consequences) then the options stop working.

This is what we're trying to do at a low-level with CodeMirror:

https://codesandbox.io/s/dank-currying-2rjxr

but it's not taking effect (either in that sandbox or our editor).

We've lost 100% height in FireFox.

Presumably down to the fallbacks for 100vh but not over the toolbar for Android in App.css. Oops.

Make serial the same dark grey as the tabs on the left.

We need to make sure we change the colour of the Xterm component itself and the background of it (that's visible at the bottom when its height is not a multiple of the line height).

It's a Chakra UI theme colour, though not sure whether Xterm.js will allow use of a CSS variable to configure it so we might need to hardcode the value if there's no better way. If we do this we should put the constant somewhere in the theme folder.

Bug Description

On iPadOS, type [] and try to delete it with backspace. You can't.

Switch tab and switch back, you can.

It seems to reproduce with almost no editor config at all:

https://codesandbox.io/s/ios-square-brackets-bug-1i55h?file=/src/index.js:394-400

but, I can't get it to happen on https://codemirror.net/6/

What's going on? We should investigate a little and then raise a CodeMirror bug.

It seems to work OK on Chrome on Android (note Android FF has some different glitches).

Environment

Amusingly I can't get the iPad to tell me what version it has because it wants to install an update, which I can't do right now.

For review branches it would be useful to deploy them even if e2e tests fail in order get earlier feedback.

We'd need to be careful any change here we didn't affect main/tag builds.

Decide the best way to handle the current file state and select newly created files.
Could just be a small change to what we have with an event for file creation or could rejig the state significantly.
Update e2e test.

See e.g. codemirror/codemirror5#6401 for the previous CodeMirror version.

This is OS-level behaviour so the only option is to work around it or live with it.

Also applies to the current Python editor. It doesn't seem to be an issue with Monaco, for what it's worth.

Consider switching to the Files view after uploading a non-main File. Alternatively (or perhaps in addition) make the confirmation dialog spell out all the changes all the time, so you know what's happening.

Some overlap with #55

We should review the accessibility of toasts vs https://www.w3.org/TR/UNDERSTANDING-WCAG20/time-limits-no-exceptions.html#essentialdef / https://www.w3.org/TR/wai-aria-1.1/#status

Different approaches probably make sense for success and error feedback. The toast currently has role "alert" in both cases.

Description

Hex and python file support, matching v2 (or discussing if there is good reason not to).

A hex-only open action already exists.

Todo

• v2 supports drag and drop to open a Python or hex file (recognised by file extension), we should add this
• revisit the open action to ensure alignment (see notes below)
• consider file system size errors for module additions
  • We just give an error on download at the moment, more work in WP4.
• e2e tests for key scenarios including those covered in v2

Implemented actions

• Open a project (files view, though I don't really the idea it wouldn't be visible by default)
• Download a hex (bottom bar)
• Drag and drop (behaves as per open project)
• Delete file
• Upload file
• Download file
• Create (Python) file
  • Missing for now: switching the editor to new files, tracked by #45.

Adding file size information to be looked at separately. For the moment you just get errors when downloading the hex.

Questions discussion

0. General feedback.

Multiple file support has been requested by accessory vendors for v2.

1. The open button and drop action has a special case for Python modules as per v2, but I think I'd like to remove it in favour of requiring the user to use the upload action. I think that the post-condition of successfully opening a project should always be having a new project open, not modifying the existing one.

Carlos strongly in favour of maintaining it as a route to open modules. Noted that it wasn't called "Open" in v2 so I've switched to "Load files" here and kept the v2 behaviour.

2. What should opening a Python file do? On v2 it leaves uploaded modules alone. I think if you've used the open action or DnD then it should replace the project entirely.

Kept v2 behaviour.

3. When do we want to confirm user actions?

• In v2, overwriting a file shows a confirmation prompt. But replacing the whole project via open or drag and drop does not. This feels inconsistent to me.

Added confirmation. For now we do it always, even if files are unmodified.

4. Given that we have support for multiple files, how do we feel about labelling main.py with the project name? Applies to the file view, and download / upload project name setting.

Dropped this for now on the basis that the existence of main.py is a bit more obvious with the Files view. Previously it hadn't been clear what "main.py" was. Might change this based on feedback.

5. Should there be a way to download your project as a whole (e.g. ZIP, or our own JSON format) that doesn't include two copies of MicroPython? We'll need some kind of package format for classroom, though it could be specific to the iframe integration.

Strong push back against this idea:

• Other file formats would cause confusion
• Might be blocked / have different difficulties working in school environments

6. Should we have an action to create a new file? A new module?

• Leaving current action for feedback.

7. Should modules be editable? What's the point in module metadata if they're just "other files" ?

• Module metadata lets us map at most one Python file without the metadata to main.py and load modules alongside it. That's its only current role.
• Leave as-is for not for feedback, but I think we should revisit before MVP. Perhaps the should be "soft read-only" so some additional action was required before you could change them.

Conclusion

Based on the feedback we made the following changes:

• Stop naming main.py as per the project. We might reinstate this depending on user feedback.
• Support loading/uploading/dropping multiple files.
• Add confirmation for all file replacements. For the moment we always do this, we could perhaps skip if files aren’t modified.
• Reorder Files area actions to match other apps (we didn’t discuss this, but I think it’s better)
• Renamed button to “Load files” as it’s only the hex case now where it actually loads a project.

So other than the main.py naming, I think it’s now as per v2 but with multiple file support and confirmation for destructive actions.
I’ve not stressed over the text in confirmation dialogs etc. as I think we can review that all at once at some point for tone/consistency.

I've also raised #43 and #44 which I'll look at shortly..

Review the image for other related changes.

No need to do any of the UI in the left sidebar on this issue.

I think the same Chakra component can be used to get the buttons in a pair as has been used with the download/flash button.

Trivial, see codemirror/dev#473

There are no compliance type links in the current design. What do we need and where should they go?

There's also no cookie notification as per classroom & .org.

The block highlighting demo is glitchy when editor width changes.

It can draw too wide and be the only reason the editor needs to scroll or too short and look daft.

1. Narrow the right hand side using the split pane
2. Make a change in the editor so it redraws
3. Widen the right hand side using the split pane
4. Note that it's not redrawn the code structure highlighting so the boxes stop short on the right hand edge.

Possible steps to investigate:

• Upgrade CM6 and see if it's still an issue. Should just be bug fixes but check nothing (additional!) is broken.
• Find out whether the view plugin is being called at all when the split pane resizes (blocks.ts)
• Find out whether CM6 is attempting to listen to its own size, if so how? Can we fix it?
• Alternatively: can we poke CM6 to make it redraw when the split pane resizes?

Bug Description

A failure in one of the e2e tests in CircleCI. https://app.circleci.com/pipelines/github/microbit-foundation/python-editor-next/217/workflows/53ef8a75-b6c6-44b6-8d6f-fa23766a93d9/jobs/225
When the workflow is reran (without any changes to the build), the failure is not there anymore. https://app.circleci.com/pipelines/github/microbit-foundation/python-editor-next/217/workflows/9f220459-f7a2-4ade-ad2e-25786aabb9e4/jobs/226

How To Reproduce

The bug happens in the e2e tests the first time running a workflow after pushing a branch for a PR.

Screenshots

Settings don't fit well in a tab and aren't of similar importance to the other tabbed items.

We should move them to be an icon near the language switcher raising a modal.

Description

The prototype editor has had code persistence but it needs reconsidering and testing.

Todo

• Discuss trade-offs and multi-tab scenarios

• Agree an approach for the MVP

  • Discussed with @microbit-carlos @microbit-giles and @daniasuheimat. We agreed that we'd go with a session-storage-only approach to avoid the complexities of other options which we documented on https://paper.dropbox.com/doc/Python-editor-V3-code-persistence--BIwME3lzfXwtQYVBw2STY5LkAg-vnbVcNeM9NRxJcAGxkiJO

• Implement and test

  • See if we can pull the version information we track into fs.ts rather than storage.ts. We don't need to persist it and it gets in the way.
  • Implement sessionStorage using base64-js
  • Give up when we hit errors, making sure we don't partially restore code (e.g. clear storage and swap in null impl)
  • Test out performance and figure out how often we ought to call it. No option to use a worker so it's got to be quick enough.

Ensure we plan for multi-file scenarios.

Conclusion

The files list isn't sorted.

We should sort it with main.py first then other files in case-insensitive alphabetical order by file name.

If we have any other places we show files we should use the same ordering (e.g. load dialogs/toast).

Description

Add a structured approach to metrics. Consider learnings from v2.

Todo

• Sentry
• GA
• Cookie notice - this is covered by the versioner.
• Integrate it for key editor actions. Detail to be addressed separately, including the extent to which we aim for comparability with v2.

I've not added our legacy metrics endpoint. We can discuss/add that in future.

Conclusion

Sentry and GA are working on our staging deployment.
Notes about first non-staging release added to #28.
We'll evolve the interface as we look at individual actions and how best to log them.

To replicate simply add large files to the filesystem and try to WebUSB flash:

The error message is probably sufficiently clear, but being wrapped in the "WebUSB error" messaging can be confusing.
It also disconnects the user from WebUSB, which doesn't need to do, as there wasn't actually a problem during the DAPJs flashing.

Review approaches to implementing internationalisation/localisation in frontend apps and prototype the most promising. Discuss with Matt and make a choice for future implementation.

Requirements:

• Must store or export/import messages in a format compatible with the Crowdin platform we use for translation. See https://crowdin.com/.
• Must support parameterised messages, e.g. "Welcome back {{username}}".
• Must have a good developer experience from React
• Must be possible to use the messages in non-React code in the same project (e.g. can get a message with an API call not just a React component).
• Must be able to load just the language data for the relevant language (to prevent bundle bloat - we'll end up with translations for 15+ languages)
• Must be sufficiently popular it's going to stick around and be stable
• Not sure how much we care about date formatting - I don't think we have any at the moment. Number formatting might be more relevant in future (e.g. graphing).

I'm open to adding to and arguing about this list. We should review all the English text in the current app to get an idea of what bits might be awkward. Prototyping shouldn't cover all the text but shouldn't stick to just the easy bits either.

It's been a while since I've looked at this space.

I've used https://formatjs.io/ (in React flavour) in the past. There's also https://www.i18next.com/. There are probably lots more!

Paper doc for investigation notes:

https://paper.dropbox.com/doc/Python-V3-translations-notes--BL0nV4c8p~G~1q~aq9D0EbPnAg-pGXKd2sTnJ9xpa6J3VZ4q

Review deferred v2 enhancements and incorporate as issues or into the broader project planning.

Bug Description

CodeMirror de-dents after an "elif" when there are no blank lines after it.

How To Reproduce

With this code snippet:

while True:
    if button_a.is_pressed() and button_b.is_pressed():
        display.show(Image.CONFUSED)
        break
    elif button_a.is_pressed():
        display.show(Image.HAPPY)
    elif button_b.is_pressed():
        display.show(Image.SAD)

Pressing enter after the last line (line 10) makes the cursor go back to the beginning of the next line (11).

Expected behavior

The curser should be indented.

Additional buggy behaviour (probably the same bug):

Giles reported: "Get to the end of typing line 4, press enter and cursor goes back to column 0 on line 5
This only occurs if there are no blank lines after."

Depends on

Other tickets:

• N/A

Third parties:

• N/A

Description

Todo

• Switch to production GA details
• Disable GA for staging
• Setup CF/DNS assuming we keep the same approach
• Double check Sentry settings + that we're happy to use it live

Conclusion

Perhaps with an interaction to make it clear it opens microbit.org.

First key features:

• Logo and width of strip on the left
• Gradient of strip on the left (leave icons unchanged)
• Box shadow on the left of the right hand side
• Button rounded corners. Use Chakra's theme to theme the button so that it looks right by default
• Colours. Does it make sense to align them to Chakra's colour scales? Get rid of any hardcoded colours, e.g. for the lines between areas. See the interaction on hover/press specified for the buttons.
• Colour toasts
  • This looks like it needs a custom toast component https://chakra-ui.com/docs/feedback/toast. Matt/Dania to look at this together.
• Attempt a version of the swerve that is grey all the way to the left but has similar but less rounded bits on the right
  • Started by Matt
• Top/bottom bar spacing @daniasuheimat
• Gray colour theme alignment @microbit-matt-hillsdon

We'll add more.

Create a variety of variants of the structure highlighting and use them as input for discussion about the feature.

Links to demos:

• https://stage-python-editor-next.microbit.org/
  • We changed the active line style so as not to conflict with the background highlights.
• https://review-python-editor-next.microbit.org/brackets/
  • The active line highlight has some issues as we've needed to make space for the bracket but it's meant it doesn't start from the gutter/left. Probably fixable but may require reimplementing the gutter.
• https://review-python-editor-next.microbit.org/boxes/
• https://review-python-editor-next.microbit.org/l-shapes/
• https://review-python-editor-next.microbit.org/lines/

Not much effort has gone into making them pretty at this stage but it seemed like a good starting point for conversation.

Please try these out by making edits to the text as well as pasting in code samples. Is it too confusing? Cluttered? Would it be better if we just provided indent guidelines every 4 spaces rather than using the AST?

How it works

All of these work in a very similar way, by using the parse tree that CodeMirror builds for the text.

Most of them are drawing a line or box for each compound statement or definition and also for the body nested within it (corresponding to indent). We're drawing the boxes where the indent should be not where it is, so it serves to indicate where indentation has gone awry.

The l-shapes one is a little different as it draws a single L shape for each "body" it encounters, drawing the L back to the parent statement.

Known issues

• As the levels of nesting increase we start to shift the boxes right a bit until it overlaps the text. Pretty confident this is fixable.
• We need to ignore comments when understanding the extent of an AST node but haven't tried to do this yet.

It's confusing to have two load/upload actions. It's pretty much impossible to label them in a way that distinguishes their behaviours.

Key differences:

1. Load has special behaviour for hex files
2. Load maps non-module Python to main.py
3. Arbitrary limitations in the "Load" (no weird extensions, mpy case)

If we decided we could:

• live without hex files in the file system
• have arbitrary file type upload
• add control of the add vs overwrite main choice

then we could collapse the two actions into one which would reduce user confusion from choosing the wrong action.

The trick is to come up with a confirmation step that doesn't feel complicated for the common case.

Description

Infrastructure setup for MVP.

Todo

• CircleCI deployment
• Setup e2e testing framework
• CloudFront

Conclusion

It's here:

https://stage-python-editor-next.microbit.org/

Review PRs will have a link to the CI build.

I've not set up the live domain yet - we can do that as part of the first release.

The deployment uses internal @microbit-foundation packages consistent with the website and classroom tool. They're only referenced in the CI script and aren't required to get the app up and running elsewhere.

There's a z-index issue with the terminal and the pop-up menu that means the menu item overlapping the main terminal area cannot be selected.

The V2 editor has a feedback form. We want something similar but distinct, linked from the new Feedback button.

1. Change program to

while True:
    print("Hi")

2. Connect and Flash
3. Open Serial (V2 only, it's already open for V3)
4. Refresh page
5. Attempt to connect

or for V2

Only physically reconnecting the device seems to fix it.

If you disconnect (or in V2 close serial) before reloading the page then it works fine.

Description

Add onbeforeunload handler to warn on quit with changes that haven't been downloaded.

We don't currently track when files have been changed ("dirty" in the old editor) but this should be easy to add in the file system code (fs.ts).

Check the situation in the real world. The text in the v2 editor is not used for me in Chrome. I think Chrome substitutes a generic dialog to prevent sites misleading the user.

https://developer.mozilla.org/en-US/docs/Web/API/Window/beforeunload_event

We could choose not to implement this if we're happy with how local storage or similar is working (not implemented as of now).

Todo

• Understand the feature in the v2 Python editor - https://github.com/bbcmicrobit/PythonEditor/
• Review the file system code and come up with a plan for tracking whether we're dirty that's equivalent to the v2 editor (but nicer!)
• Add a listener somewhere appropriate
• See if we can e2e test this. Can we interact with the dialog via puppeteer? If we can then I suspect the v2 e2e tests will have an example.

Conclusion

If we stick with a bottom toolbar then the keyboard on iPad is problematic. On Android the viewport resizes. On iOS it's overlapped.

There is an API that we can use to detect and resize, but it's not possible in CSS:

https://tkte.ch/2019/09/23/iOS-VisualViewport.html
https://github.com/adamjgrant/a-viewport-that-works/blob/master/js/viewport-fix.js

Not clear whether this would result in satisfactory UX.

Description

The prototype has WebUSB support based on the the v2 editor.

This needs to be reviewed and update to provide at least as good a quality experience as the v2 editor.

Todo

• Collect input on the current implementation.
• Review and test the existing code. Structure the testing with a view to extracting the code to a separate library in future.
• Sync with recent changes in the v2 editor.
• Write or repurpose manual tests for scenarios that cannot be automated (e.g. unplugging the device).
• Bring UI for showing flashing progress, success and error handling up to a sufficient standard for MVP.
• Reinstate the event / metrics tracking. The actual reporting can be via a stub to be tackled separately.
• Plan any future work

Conclusion

This can have pretty poor consequences for the UI.

I tried the obvious fix with no joy so it needs a more detailed look.

It's annoying to try to select code and end up selecting line numbers.

You can only really get it wrong at the moment if you try to start your selection from the left and miss but I still think this might be worth it.

We could just prevent line number selection entirely with user-select: none ? This seems to be what Visual Studio Code does. I've never missed trying to copy them in VS Code.

Screenshots used in the README. Keep the old ones for the entertainment value (newest first).

On #143 I hit an upgrade issue with Chakra UI 1.6.3.

There's a repro here
https://codesandbox.io/s/upbeat-solomon-6vttf?file=/src/App.tsx

Though it made me question whether we need the portal at all so I haven't raise it yet.

There's also the possibility of the isLazy prop on the menu as a workaround (untested).

Demo some monospace fonts.

• Jonny wants us to try Adobe Source Code Pro
• Carlos wants an Ubuntu one
• Magda likes the CodeMirror one we're using (but not the one in serial)

Depends on

#13

Description

Extract the WebUSB code and associated tests to its own library.

Todo

• Review and generalise the interface. Get input from the micro:bit team.
• Split out and setup Git, CI etc. for separate project.
• Add documentation suitable for an external audience, including a vanilla JS demo.

Conclusion

Bug Description

Code structure highlighting disappears in the "elif" block of an if statement.

How To Reproduce

Steps to reproduce the behavior:

1. Example code: https://microbit.org/projects/make-it-code-it/magic-8ball/?editor=python
2. Paste the code up until the second if statement:
   
3. Add a few blank lines after:
   
4. Go back to the end of the last non-empty line, press enter to go to a new line, and type "elif". It should still be highlighted at this stage:
   
5. Press space right after the "elif":
   

Parsed tree (logged to Console with its toString() method) before adding the elif and pressing space:
blocks.ts:97 Script(Comment,ImportStatement(import,VariableName),WhileStatement(while,Boolean,Body(":",IfStatement(if,CallExpression(MemberExpression(VariableName,".",PropertyName),ArgList("(",String,")")),Body(":",AssignStatement(VariableName,AssignOp,CallExpression(MemberExpression(VariableName,".",PropertyName),ArgList("(",Number,",",Number,")"))),IfStatement(if,BinaryExpression(VariableName,CompareOp,Number),Body(":",ExpressionStatement(CallExpression(MemberExpression(VariableName,".",PropertyName),ArgList("(",MemberExpression(VariableName,".",PropertyName),")"))))))))))

After:
Script(Comment,ImportStatement(import,VariableName),WhileStatement(while,Boolean,Body(":",IfStatement(if,CallExpression(MemberExpression(VariableName,".",PropertyName),ArgList("(",String,")")),Body(":",AssignStatement(VariableName,AssignOp,CallExpression(MemberExpression(VariableName,".",PropertyName),ArgList("(",Number,",",Number,")"))),IfStatement(if,BinaryExpression(VariableName,CompareOp,Number),Body(":",ExpressionStatement(CallExpression(MemberExpression(VariableName,".",PropertyName),ArgList("(",MemberExpression(VariableName,".",PropertyName),")"))),⚠)),⚠)),⚠)),⚠(elif))

Expected behavior

The "elif" and what follows are meant to be highlighted as well.

Status

We simplified this a bit and raised codemirror/dev#487

Bug Description

A clear and concise description of what the bug is.
A red "CR" mark is present at the end of every line in the text editor of the Microbit Python editor, when the project is built locally on a Windows machine.

How To Reproduce

Seems to only happen when using a Windows machine, after building the project locally and opening the localhost.

Expected behavior

Not have "CR" at the end of every line.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

Desktop (please complete the following information):

• OS: Windows 10
• Browser: Chrome
• Version: 89.0.4389.128

Merging #102 seems to have broken the e2e tests in CI.

This branch from just before consistently passes but we've repeatedly failed on main:

https://app.circleci.com/pipelines/github/microbit-foundation/python-editor-next?branch=ci-tests

https://app.circleci.com/pipelines/github/microbit-foundation/python-editor-next?branch=main

The tests pass for me locally with and without a production build. Something odd is going on.

Depends on

Related to #19 but we can try things out here without blocking on that.

Description

The various prototype layouts use react-spaces, which was easy to get started with. It, or our use of it, has some issues:

• Not touch friendly. Need to look for good examples here
  • You can make the touch target bigger. Maybe we can build something with that?
• Can get in weird error states where you can pull UI so far off screen you can't get it back
• No obvious path to collapse/expand support
  • I've done something along those lines for serial, but I can't see a path to animate it.

Consider also the potential need to do different layouts on tablet.

Todo

• Investigate and demo alternatives
• Switch to something better aligned with our needs

There's not much code here, it's just the main workspace layout.

Conclusion

We should consider adding some friction to editing modules with magic marker to avoid accidental changes.

E.g. the top bar could show an "Edit" button.

Known issues:

• Last line doesn't get indented (not considered part of Body?)
  • This has been fixed
• Giles reports: "I pressed enter at the end of an indented line, puts cursor at an indented position but I wanted to go one level back, backspaced, typed ‘else:’, pressed enter again (not the last line, there are blank lines following) and new line has cursor at column 0"
  • This is still an issue but doesn't happen at the end of the script.
• Missing dedents for pass and return etc.

Carlos:

Useful reference:

https://github.com/microsoft/vscode-python/blob/main/src/client/language/languageConfiguration.ts

The dedent else controversy: microsoft/vscode-python#6886

We should review best practice from elsewhere. Up/down keyboard navigation and something to communicate the nature of it to screenreaders would make sense. Lots of the other examples in similar apps are trees, but we probably don't need that complexity. Perhaps it is just tabs but that feels confusing.