Code Monkey home page Code Monkey logo

flipmouse's Introduction

FLipMouse

The FLipMouse (a.k.a. Finger- and Lipmouse) is a replacement for a normal PC mouse / keyboard / joystick. Instead of moving the mouse device with your hand and clicking with your fingers, the FlipMouse can be controlled by applying very low forces to the mouthpiece (joystick) with your lips, fingers or other body parts. The clicking functionality can be accomplished by sip- and puff-activities into the mouthpiece or via external switches. All settings and functions of the FlipMouse can be tailored to specific user capabilities or needs. Multiple settings can be stored into the device and changed via desired user actions. Additional features like built-in environmental control via infrared, optional bluetooth-add-on module for controlling smartphones/tablets or complete software-based control of all functions via serial command interface make the FlipMouse one of the most flexible alternative input devices available today.

FlipMouse with package and mouthpiece

The FlipMouse might be used as a full replacement of standard computer input devices and can also be used for accessing smartphones or tablets (via the standard HID support or accessibility features). It can be useful for people with motor disablities, computer gamers, musicians or people who want a hands-free computer access for other purposes.

Our goal is to provide an affordable DIY-solution for everybody who wants to use a PC or smartphone with non-standard interaction methods.

Please note: There are different versions of the FLipmouse hardware and software! This Github repository refers to the current Version 3. If you are looking for Version 2, use this repository: FLipmouse V2.

The DIY construction kit

The official FlipMouse DIY kit is available for purchase via https://asterics-foundation.org. We provide a Construction Manual which shows how to assemble the kit. In case you want to buy the components from other sources, a full part list is available in folder Hardware. The design files for the enclosure can be found in folder Hardware/case-design. The PCB designs (schematic and layout) have been made with KiCad and are available in folder Hardware/PCB-design

Hardware and Features

The FLipMouse uses an Arduino Nano RP2040 Connect microcontroller (ARM CortexM0+ architecture) as main module. Movements of the mouthpiece are measured via Strain Gauges (DMS), which detect small forces applied to the mouthpiece / joystick. The sip/puff actions are detected by a pressure sensor. Via two 3.5mm jack plugs, external momentary switches can be connected. From all these user inputs, desired actions can be assigned via the FlipMouse WebGUI, for example mouse/keyboard or joystick actions via USB HID. Via bluetooth, BT-enabled devices can be paired and mouse/keyboard functions can be used wirelessly. Furthermore, an IR-receiver module and a IR-LED allow recording and replay of arbitrary infrared remote control commands.

Software

The FLipMouse firmware is based on the Arduino framework. The firmware implements a composite USB HID device (mouse, keyboard, joystick and a serial port in one device). The mouse and keyboard device classes are used to transmit different keys or mouse actions to the host device. The serial port is used configure the FLipMouse (or even use it as a mouse simulator via AT commands). Multiple configuration settings can be saved (stored in an EEPROM module) and changed via desired user actions. For more information about (modifying) the FLipMouse firmware see https://github.com/asterics/FLipMouse/wiki/dev-firmware Please note the COM-Port installation instructions for older Windows systems (Win7/8) here.

Configuration Manager

All settings of the FLipMouse can be changed via the FlipMouse Configuration manager
Using the configuration manager in the Chrome/Edge web browser, you can assign different actions to all hardware inputs of the FLipMouse (mouthpiece, sip and puff, external buttons,...). The Configuration manager can also be used to update the firmware of the FlipMouse and the optional Bluetooth module.

More Information can be found in the user manual: user manual (english version), user manual (german version).

Building the firmware

In order to build the firmware following prerequisites and dependencies must be installed:

  • the Arduino IDE (v1.8.16 or above)
  • the arduino-pico Core (v3.0.0, can be installed using Arduino IDE's Arduino board-manager)
  • the Adafruit Neopixel library (v1.11.0, can be installed using Arduino IDE's library manager)
  • the NAU7802-DualChannel library (dual channel patch by Beni Aigner)
  • the LoadcellSensor library for sensor signal processing (by Chris Veigl)
  • the SSD1306Ascii library by Bill Greiman (v1.3.5, can be installed using Arduino IDE's library manager)
  • select the board 'Arduino Nano RP2040 Connect' in the Arduino IDE tools menu, and the Flash Size "16MB (Sketch: 15MB, FS: 1MB)"
  • select the correct Port after connecting the Arduino to your system
  • compile and upload the firmware

Cleaning and Safety

IMPORTANT: If the mouthpiece is exposed to saliva, please clean/replace the mouthpiece (or its filter) on a regular basis, see Safety Instructions

FlipMouse

Links and Credits

Most of the work for the FLipMouse has been accomplished at the UAS Technikum Wien in course of the R&D-projects ToRaDes (MA23 project 18-04) and WBT (MA23 project 26-02), which have been supported by the City of Vienna, see: ToRaDes Project Information, Webpage WBT project.

Have a look at the AsTeRICS Foundation homepage and our other Open Source AT projects:

  • AsTeRICS: AsTeRICS framework homepage, AsTeRICS framework GitHub: The AsTeRICS framework provides a much higher flexibility for building assistive solutions. The FLipMouse is also AsTeRICS compatible, so it is possible to use the raw input data for a different assistive solution.

  • FABI: FABI: Flexible Assistive Button Interface GitHub: The Flexible Assistive Button Interface (FABI) provides basically the same control methods (mouse, clicking, keyboard,...), but the input is limited to simple buttons. Therefore, this interface is at a very low price (if you buy the Arduino Pro Micro from China, it can be under 5$).

  • FLipPad: The FLipPad controller: a flexible touchpad for controlling computers and mobile devices with minimal muscle movement.

  • AsTeRICS Grid: Asterics Grid AAC Web-App: an open source, cross plattform communicator / talker for Augmented and Alternative Communication (AAC).

Support us

Please support the development of FLipMouse by donating to the AsTeRICS Foundation:

   

flipmouse's People

Contributors

benjaminaigner avatar chrisveigl avatar deinhofer avatar fabiansteiner avatar gafert avatar junaidwien avatar klues avatar ma0027 avatar sabicalija avatar slehrbaum avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flipmouse's Issues

Couple of unclear instructions in Construction Guide

Hi,

a couple of proposed amendments to the construction guide:

  • It would be prudent to make the user check values or text of the resistor banks before soldering them in as they're easily confused. There's a warning text "Take care of the text on these parts" but that's neither English nor does it give any hints what the text should be. A warning to the likes of "Before soldering make sure you're using the correct resistor bank with a value of x Ohms, verifiable with a Multimeter or by the text "xx" on the part"
  • Maybe a warning to check the continuity / proper connection of the soldering points on the bottom of the board befopre soldering the pressure sensor because it covers those pins and you can't check afterwards which causes debugging headaches
  • On the instructions for soldering the pin header on the daughterboard, nowhere does it mention that there's a little notch that should be aligned with the position of the header and on the illustration it's also not there - this is probably the most important of the points. Furthermore, the instruction "bend the pin header to a straight angle" is a bit confusing - you might mean "making it level/even with the board"? This isn't as big a problem because the image makes it pretty clear how it should look.

I understand that a lot of these issues are problems that only a non-experienced constructor might encounter, but in the spirit of keeping the project easily accessible by anyone and the otherwise very good Construction Guide, I personally think these should be fixed to prevent people making the same mistakes I have. Maybe I've just missed these things in the construction guide, in which case feel free to ignore this.

Thanks

GUI: losing slot settings if not every slot is stored

If the settings for one slot are not stored and you switch to another slot to configure, the settings get lost.

situation:
4 different slots

goal:
change button 2 on all slots to another action

steps:
select one slot, change the setting. Load next slot and change that (for every slot)

problem:
the changes are only stored for the last slot.

workaround:
Press "Store settings" each time you changed one slot

openinput support

This could be a very interesting target to add to the openinput project, creating this issue to check interest and show my willingness to work on support for it.

Add AT command to switch on/off certain buzzer beeps

As requested by a user, we could implement a command to switch on or off different kinds of tone outputs:

  • sip / puff
  • enter or exit strong sip / puff
  • calib
  • change slot
  • IR receive / send
  • BT pairing

Suggestion:
"tone off"
AT TO 0011001100
(each bit indicates one tone type)

EDIT:
we should implement this for this repository as well as FMv2

Inconsistent EEPROM state

When programming a FLipMouse, normally there are no slots available (only the default config, not saved in EEPROM).

Using a terminal program, entering:
"AT" --> "OK"
"AT LI" --> no feedback
"AT LA" --> should write the default slot,but I get following incomplete feedback, error messages and a stuck FLipMouse:

Initializing EEPROM!
Deleting all IR slots
Error: seperator problem0
Error: seperator problem0
Error: seperator problem95
Slot:
AT AX 40
AT AY 40
AT DX 20
AT DY 20
AT MS 50
AT AC 20
AT TS 400
AT TP 600
AT WS 3
AT SP 800
AT SS 10
AT MM 1
AT GU 50
AT GD 50
AT GL 50
AT GR 50
AT RO 0
AT BT 1
AT II 
AT BM 01
AT 

Current master (v2.8.3).
It is also not working with the C# GUI.

Infrared: record sometimes fails

If you record a remote, sometimes only 1 edge is recorded.

Of course, there is no IR command with 1 edge.

There should be some plausibility check of the recorded signal

Toggle click hold

A command should be added which does not simply hold the mouse click, but toggles hold by the same button.

3d-printed case: improvements

labels/tactile indicators for the buttons (at least B2 / B3) would be nice
and we should improve the 3d-print so that the M2x16 mm screws / bolts have metal nuts or a better fit for the threads

Automatic Firmware Update

it would be great if we could enable automatic firmware updates for the BT-Add-On from the GUI (as it exists already for the Teensy firmware)

Theoretically it could work if we replace the second GND signal from the AUX connector with a connection to GPIO0 of the ESP32 and add a way to reset the ESP32 (maybe using an analog circuit)?

Then, we could forward the serial communication during the FW-upload from esptool to the ESP32 via the TeensyLC....

Keycode lookup if esp32_mouse_keyboard aka ESP32miniBT module is connected.

Currently, it is not possible to use different keyboard layouts than US for the FLipMouse.
As we have limited resources in the FLipMouse (current master: 92% Flash, 79% static RAM usage),
we could implement a lookup of keys / text or keycodes via the ESP32 module (4MB flash -> we can store all supported keymaps of Teensyduino there).

In my opinion, we need following new features:

  • Detect if a compatible ESP32 module is connected
  • A dynamic table which contains the most recently used keycodes
  • Update mechanism to load a new keycode for a key from ESP32
  • Fallback solution (e.g. currently used static keymap) if dynamic update is not possible

Extra hole for external Sip/Puff

An extra hole should be on the bottom of the FLipmouse.

In the case of using the FLipmouse as finger operated mouse, it should be possible to route the pipe to the outside

`AT SR`: add button states

I think it would be an improvement to see current button states (physical & virtual) in report raw values.
We could visualize states in the GUI and process this data otherwise.

@ChrisVeigl Do you think it would be a problem to just add another field in the report?
E.g.: <pressure>,<up>,<down>,<left>,<right>,<x>,<y>,<buttons>

Current C# GUI does not support this format, no live values are shown anymore.

FlipMouse C# GUI - Problem if ID string is assigned as keyboard write

If at any button, the command "Write Text" is assigned and this text contains the word FLipMouse, it will be parsed as ID String.
This results in various errors:

  • If the version is not in this string, the GUI asks for firmware updates
  • The notification for loading the FLipMouse config is shown again.

The only way to avoid this error is clicking "no" if the GUI asks for loading the config from the FLipMouse.

FLipMouse GUI - Managing Bluetooth devices

Currently, the FLipMouse with ESP32miniBT addon accepts every pairing request and paired devices can't be unpaired.

It would be nice to show (if available) a list of bonded devices in the FLipMouseGUI (C# & Web), including the possibility to remove one or all devices from this list.

Media keys

Teensyduino supports media keys (e.g. play/pause, next, prev.,...) with the currently used Keyboard.press() and Keyboard.release().

We should add this functionality in:

  • FLipWare (keys.cpp)
  • FLipMouseGUI
  • FLipMouse WebGUI
  • esp32_mouse_keyboard

Flipmouse GUI: auto-increase targets wrong slider

If configuring the cursor movement (first tab) in split X/Y mode,

the "+" symbol of the speed Y slider increases the X value, but only while auto-increasing via hovering.
If you click on the button, the correct value is increased

Saving config: performance and usability

Usability

Current Workflow

The current workflow for making configuration changes is:

  • Open GUI
  • Make changes in GUI
  • Click "Test settings" in C# GUI, WebGUI automatically "tests" settings
  • Click "Store settings" (C# GUI) / "Save" (Web GUI) in order to persist the configuration

Alternative Workflow

This would be an alternative workflow:

  • Open GUI
  • Make changes in GUI
  • Changes are applied and persisted immediately, no further user actions are required
  • There are possibilities to revert the configuration to default configuration or the starting configuration of the session.

I think most of the time the user wants to keep the new configuration, so it would be a benefit for usability if no explicit action would be required for saving the config. It's also a benefit for testing if changes (e.g. sensitivity) immediately take effect and there is no need for clicking on "Test" (currently already implemented in WebGUI).

Technical stuff

Currently it seems to be only possible to persist a config like this:

  • delete all slots with AT DE
  • re-save all slots
  • this takes about 4 seconds

Proposed changes

  • AT SA <slotname> should overwrite an existing slot with the current config, so a single AT SA command would be enough for persisting -> some milliseconds instead of 4 seconds
  • I would implement WebGUI with the alternative workflow, C# GUI should stay like it is.

@ChrisVeigl, @benjaminaigner: what do you think?

Sip&Puff sensor spurious failing

@ma0027 had the problem of a failing sip&puff sensor.

Workaround: unplug & plug in for a few times.

Reason: unknown

Maybe we should add error handling for the sensor & a restart procedure of I2C.
In addition, it would be good to have error output and maybe a "storage" of error messages.

Optimize EEPROM structure

Currently, the settings/slots are stored in fixed sized windows.
Improve the storage to optimize enhance the memory space layout.

Flip/mirror X/Y axis

it should be possible to flip or mirror the X/Y axis to enable more flexibility for mounting (mounting in all rotations)

BLE: rate limiter / commands to addon module in general

For adapting the rate limiter, a new AT command needs to be implemented.

But in general, I would prefer to use a general AT command, which is used to send arbitrary commands to an addon board.

Suggestion: AT BC <command (according to asterics/esp32_mouse_keyboard#39
e.g., AT BC $RL80 sets the rate limiter of the esp32 addon to 80ms.

In general, these commands can/should be:

  • sent from the GUI
  • sent from a WebGUI
  • stored in the EEPROM (we should agree on a procedure how an array of external commands can be saved in EEPROM).

Improve batch download of slots

Currently, the GUI saves all slots in one batch, all with an "AT" command.

Maybe it should be possible to switch of the preceding "AT" for a short time during the batch download.
This would decrease the sent bytes by maximum 50% (for each command without a parameter)

HW - evaluate best USB plug positioning

In many cases the USB plug on the back side might not be the best solution.
For the new hardware design, we should evaluate the best possible position of the USB plug

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.