Invisible Pen — make any screen a touchscreen without special hardware

a teacher creating a projecting her handwritten notes by placing her phone on an elevated platform
Credits: https://www.linkedin.com/posts/dishagugnani_teachers-learning-teaching-activity-6705069147409174529-UMur
A student using a compact disc as a mirror to project their keyboard over a zoom video call
Credits: https://www.linkedin.com/posts/sarthak-agarwal-04971b68_just-wanted-to-share-with-you-it-may-help-activity-6709076668629176320-ZeJ8
another teacher writing on a mirror with a bright marker so that students can see both his face and what they are writing
Credits: https://win-vector.com/2020/11/05/new-free-mini-lecture-simpsons-paradox-in-a-b-testing/
  1. lets the presenter use their hand (and not mouse or external hardware) to express their ideas
  2. uses existing devices in houses so that it is economical
  3. is easy and free for anyone to use and setup
  4. should work always i.e. is reliable
  5. should be accurate enough to be usable for the above problem
Invisible pen setup
  • The mobile device which is mounted on the tripod tracks the live hand movements of the presenter / teacher
  • A server on the laptop receives the coordinates of the index finger from the mobile device over a local secure connection
  • The server translates the finger coordinates to pixels on the screen and moves the mouse pointer thereby tracing the presenter’s drawing on the screen

As we directly control the mouse, this setup can be used beyond rough sketching use cases — e.g. flip the pages of an e-book or a PDF document

Tech Stack

  • On-device hand tacking is done by handpose, as a Tensorflow.js model
  • Front end business logic is written in React.js & Next.js
  • Predicted hand coordinates are sent wirelessly to a Python Flask web server using socket.io, which is on the same local network
  • Our custom translation logic written in Python converts hand coordinates to screen pixel coordinates
  • pyautogui and pynput Python libraries control the mouse and keyboard of the laptop

On-going work

  1. Model accuracy
  2. On-device model performance
  3. Optimize the communication channel between the phone and laptop
  4. Robust computer algorithms to translate hand coordinates to the laptop’s screen pixels more accurately
  5. User experience research to improve the setup and onboarding
  6. Other mobile configurations besides behind the human, currently shown above

--

--

--

Applied Deep Learning Engineer | LinkedIn

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

ARC Explained

Using the new D8 core scaffolding tool

Escher — The world’s first cloud-controlled Etch-a-Sketch

Future of Mobile Apps

Basic Builds :: How to update data in a Shiny App on RStudio Connect

Android, Kotlin and API 27

CS371p Spring 2022: Daniel Cai

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nitin Pasumarthy

Nitin Pasumarthy

Applied Deep Learning Engineer | LinkedIn

More from Medium

Confession to Open Source

Why automated testing is so important on videogames nowadays

AngularJS End-of-Life is Here: Now What?

Open web learning sessions

Writing WWW with a pencil