This dynamicland demo shows how the system allows colored centimeter cubes to be detected for use as flexible inputs.

For example, they can be placed on the end of a wand to control some parameter by position.

In setup I'd like to input the projection surface dimensions and resolution. Then we could make drawing utilities to draw in real-world units.

As seen in Dynamicland, having keyboards in the space that can be used to edit code using "editor papers" is key. Some ideas for how we can do this:

• Multiple keyboards connected to one computer, and then write some native program that somehow sends events to the browser (how?).
• Each keyboard has its own little computer attached to it (which computer and how?), which then sends events to the paperprograms server, which then sends those events to the browser (will the latency be low enough?).

Right now if a corner was previously detected but then disappears, we assume it's in the same location as before. Instead we could remember distances+angles between corners so moving papers is smoother.

Dynamicland has bluetooth keyboards that can be pointed at programs to allow in-space editing. Example video below:

I think Operating Systems always assumes one keyboard, so to allow multiple keyboards to modify programs in parallel probably requires reading bluetooth signals from each directly.

• Web Bluetooth API
• Interact with Bluetooth devices on the Web

(Still unsure about many parts here. Just creating a place for brainstorming)

When trying it out with people it was confusing that set sets data for only one paper, but then getting data works for all papers.

Now that threshold and especially sigma might come in, it's becoming annoying.

Something where on boot, it tells you to put a piece of paper down, and it searches for parameters such that it can recognize that piece of paper?

idea

I would like to propose an API for papers to communicate with each other by sending events.
The idea came from a usecase like in this tweet - basically a setup in which one has a keyboard-paper that emits events (e.g. keypress) and sends them to another paper, which can then handle those events.

the API

The API would look similar to that of the Node Event API. A paper could emit events like so:

/* paper.emit(eventname, paperNumber, data) */
paper.emit('keypress', '123', {key: 'a'})

with the second parameter specifying the paper to which you want to send the event. Alternatively one could also have a broadcast method if one wanted to send an event to all papers:

paper.broadcast('keypress', {key: 'a'})

and handling of events would look like so:

paper.on('keypress', function(data) {
  // do something
})  

Any opinions on that?

Hello,
Please, what is the current state of the project? I've been really excited about this program and have been telling people about it for the last two years. I was looking at the Fork Network and vedran and cesmoak have both done substantial work that hasn't been merged into janpaul123's repo. Does anyone else have a desire to see paper programs continue? I've read #32. I think Paper Programs would be a great education platform for kids. I'd like to see it in our local library or in houses with homeschoolers. I'm planning on visiting Dynamicland eventually, but a project that can be installed is a lot more accessible to people that can't afford the plane ticket. 2018 was two years ago now and I haven't seen any new content up on their website.

So is there anyone else who would like to continue Paper Programs?

This is a bug in Chrome and is being tracked here: https://bugs.chromium.org/p/chromium/issues/detail?id=799362

This is the page to reproduce: http://janpaulposma.nl/offscreen-canvas-bug/

I wanted to experiment with multiple whiskers but creating the second whisker failed with the error:

Error[program:0:0]: "Cannot read property 'getContext' of null"

The problem seems to be that paper.get('supporterCanvas') only returns a reference to the canvas on the first call. I assume the reason for this is that you can call transferControlToOffscreen only once. Instead of just returning null I think it would be nicer if paper.get('supporterCanvas') would cache the reference to the canvas on the worker side and return the cached reference instead. I think this would be generally useful independent of the whisker issue. To improve handling multiple whiskers I would like to change the current implementation so only a single setInterval is created by paper.whenPointsAt().

If we adopt the "claims and wishes" system in Paper Programs more and more of the code should be moved to actual papers. I feel slightly uncomfortable storing all my code in the database. I think it's better to store the files on disk. That way it's very easy to backup / version code. It's also easy to browse the state of the system and use any existing coding tools if necessary. It would also make sharing code digitally more easy. Right now I see a usage pattern where I write code inside paper programs and then also copy it to a file so I can share it with others and have it savely stored.

I propose the following folder structure where each space has its own subfolder modified programs have two files the original file with the name {number}.js and _{number}.js for the modified version which would be actually executed.

-- programs
    |
    -- space1
    |     |
    |     -- 1.js   // printed version
          |
          -- _1.js // current temporary version prefixed with 

What are your thoughts about this?

The webcam I'm using (Microsoft LifeCam 3000) doesn't work with Paper Programs. The "onloadedmetadata" event never get triggered. This problem unfortunately only occurs with Chrome (65.0.3325.181), other browsers work fine. The cause seems to be how getUserMedia is called:

If I change it to a specific resolution, my webcam supports it works. I think this is a bug with Chrome or the webcam driver.

video: { 
  width: 1280, 
  height: 720
}

I searched for a more generic solution which always picks the best available resolution and found this snippet on stack overflow. This also works with my webcam.

video: {
  optional: [
    {minWidth: 320},
    {minWidth: 640},
    {minWidth: 1024},
    {minWidth: 1280},
    {minWidth: 1920},
    {minWidth: 2560},
  ]
}

https://stackoverflow.com/questions/27420581/get-maximum-video-resolution-with-getusermedia#answer-27444179

Is this issue specific to my webcam or has anyone experienced something similar?

When light is projected onto the keypoint dots, it can disrupt their detection, causing the program to disappear.

Dynamicland has recently fixed this by projecting shadows (black circles) over each dot to prevent interference from the projector light:

I'm thinking just a single supporter canvas will draw black circles over each detected point—with a high z-index to appear above all other canvases.

I'm seeing framerates that are either 1 or zero depending on the size of the polygon I draw on camera.html. This is using Chrome Dev Beta Version 66.0.3346.8 on a MacBook Pro (Retina, 15-inch, Mid 2014). Camera is a Logitech BRIO via USB.

I ran some basic profiling in Chrome and as expected it looks like a lot of time is spent in the compiled opencv code.

I'm wondering if I need to write some code that uses WebGL to analyze the frame or if there's another simpler approach here that will work.

Has much effort been put into performance yet? Any low hanging fruit I can help investigate?

Should we implement some of it? Or not? See more discussion here #32

Currently if you pass an unknown command to paper.get it blocks the program indefinitely.

const number = await paper.get("paperNumber") // typo, "number" would be correct

Instead I would propose to throw an error in this situation like:

Unrecognized command paper.get("paperNumber", ...)

Since Paper Programs may have a lot of growth left to do, it might not make sense to write too much documentation. I think for now it would be enough to distill the video tutorial into a text-and-pictures markdown document, and maybe have one or two documents explaining the architecture in more detail than the README.

Given that this would be a small effort that would probably be a big help for new developers, I'm thinking of taking this on first.

In a project this complicated (i.e. one which relies on hardware, real-world lighting, browser features hidden behind flags, etc.) I imagine it's difficult to write useful unit tests in any conventional way. However, I also think it's really important that developers have a couple smoke tests to validate that they aren't breaking things when they check in code. I can think of a few steps that would help make the code more testable:

1. Break the call to navigator.mediaDevices.getUserMedia() out of CameraVideo and instead pass a stream as a prop.
2. In general, divorce the openCV-related stuff from the React components and just pass the components the minimal state they need to render. That state could be managed by something like Redux, although in my experience Redux itself is not always a good fit for apps whose state includes "blobby" things like MediaStreams.
3. I haven't dug too deep into the server side, but if we could find a way to mock the DB (or just the calls to the DB) it would make it easier to test changes to the API.

Given that Paper Programs is still young, there might be other higher priorities right now. I just wanted to throw this out there as something to think about while moving forward or designing new parts of the system.

I'm not sure what I'm doing wrong, but the new camera api is not working for me.

This is my paper program:

// Cam
importScripts('paper.js');

(async () => {
  var canvas = await paper.get('canvas');
  var ctx = canvas.getContext('2d');

  setInterval(async () => {
    const papers = await paper.get('papers'); 
    const camera = await paper.get('camera');

    paper.drawFromCamera(ctx, camera, papers[2068].points, papers[757].points);
    ctx.commit();
  });
})();

Initially I got the error that the data is not transferable:

I'm wondering why this line doesn't have an await even though ImageCapture.grabFrame returns a Promise. Is that a mistake or am I missing something?

When I add a await I get a different Error from ImageCapture.grabFrame:

@shaunlebron Can you give me some pointers what I'm doing wrong. Thanks, I'm really excited to build something with the new camer API.

As lighting conditions change, I find myself having to adjust the camera exposure all the time. It woudl be great to automate this. Unfortunately we cannot (yet) do this from the browser, so we might have to find some native app or write something small ourselves.

https://www.npmjs.com/package/get-port has a neat way of doing this - you request 3000 but if that doesn't work, it grabs another one.

You can then use https://www.npmjs.com/package/opn to automatically open it

And then in the browser, grab the port from window.location

Lemme know the good/bad/ugly of this idea

Example:

// Program 7
importScripts('paper.js');

(async () => {
  var i = 0;

  setInterval(async () => {
    console.log(i++);

  }, 2000);
})();

Log output in editor:

console.log[program:9:13]: 0
console.log[program:9:13]: 0
console.log[program:9:13]: 0
console.log[program:9:13]: 0
console.log[program:9:13]: 0
console.log[program:9:13]: 1
console.log[program:9:13]: 1
console.log[program:9:13]: 1
console.log[program:9:13]: 1
console.log[program:9:13]: 2
console.log[program:9:13]: 2
console.log[program:9:13]: 2
console.log[program:9:13]: 3
console.log[program:9:13]: 3
console.log[program:9:13]: 4

I think the flushing of the log in the editor isn't working correctly but I haven't confirmed that yet.

I found a bug when requesting multiple canvases like this:

 var canvas2 = await paper.get('canvas');
 var canvas = await paper.get('canvas', { number: 42 }); // some other paper number

I get the following errror

The cause is that the canvases have duplicate keys which leads to bugy behaviour.

Interestingly it works when requesting the canvases the other way around:

 var canvas = await paper.get('canvas', { number: 42 }); // some other paper number
 var canvas2 = await paper.get('canvas');

I've fixed the issue by generating unique keys.

Here's something that I've been thinking about contributing, but would like to gauge how others feel about it before implementing it.

Right now all programs are saved to local storage, which has the downside of tying your programs to one device. This is fine if you have a single (portable) computer, but I've already encountered a few scenarios where I'd like to load programs on a different machine and been faced with copying and pasting code over, and having to re-print all the programs.

I also feel like it'd be best if a feature like this would be out of the way and limited in what it can do so as to keep the emphasis on the physical sharing of code. It'd really just be for the scenario of 'I want to plug in the projector and camera into a different machine but keep the programs I have'.

Here's what I was thinking: Add a new page at something like /storage.html, which presents two options:

• A 'Download programs' button that would export the program code and the current space's program IDs in local storage to a zipped JSON blob.
• An 'Replace programs' input that would take one of these archives and use it to replace all the program code and current space's program IDs in local storage.

Would this work as I think it would, or am I overlooking something? Would this be useful to others?

Instead of color coding programs, do you think it would be possible to encode them by using different orthogonal, distinct shapes and fill patterns as an alternative, e.g. solid squares, empty square, triangles, upside-down triangles, plus symbol, and so on?

That would lower the entrance barrier to those who only got a monochrome printer at home.

PS. Thanks for sharing this and making it open source. I've stumbled upon the Dynamicland website and Twitter feed, but failed to get ahold of DIY info or source code, so I was happy to see yours.

I spent a few minutes typing this up for our group today with the goal of making the API docs more skimmable:

https://gist.github.com/shaunlebron/27b2339048c8d1746bd1b09ccf52ea7d

Would this be a better format for the API docs, or are there plans for something else?

Anyone else hung up on getting consistent dot detection? No matter what lighting conditions & webcam settings I try I'm recalibrating every few minutes :(

current plan to redo #38 with correct transformation

User-facing:

• program must request the most recent camera image from main thread:
  • const camera = await paper.get('camera')
• program can draw image regions from the camera onto the supporter canvas:
  • drawFromCamera(supporterCtx, camera, srcPoints, dstPoints)
• program can use existing points property on each paper to retrieve/draw images over them:
  • drawFromCamera(supporterCtx, camera, otherPaper.points, myPaper.points)

Internal:

• camera image
  • in order to pass an image via postMessage, we may have to render video element to a canvas first
  • render an image from video element at most once per frame if any programs are requesting 'camera', then send the image to each.
• drawFromCamera utility function in paper.js
  • points format should either be an array of four or three points, or {topLeft,topRight,bottomLeft,bottomRight}
  • should project the srcPoints using forwardProjectionMatrixForPoints(config.knobPoints) to convert our canvas coords to camera coords.
  • will need to check the texture mapping code here to see if we need to perform a similar operation for dstPoints

This project's use of webpack's multi-entry point feature is awesome. I think organizing each entry point's modules into a folder (i.e. a /camera folder containg all the code used in entry-camera.js) would help make it clear which code is used on which page. Also, it might be a good idea to pull all the JS code from the /www directory into /client just so that all the JS code is in one place.

When I follow the docs and attempt to calibrate a color by clicking on the indicator in the side panel and then clicking on the appropriate dot picked up from the calibration page, nothing happens.

I can see by looking at LocalStorage that nothing is being set and keyPoints here is always zero-length.

Others seem to be having no problem but this has me stumped. Is there anything obvious I could be missing?

When trying to run locally I'm getting this error when running npm run dev:

createdb: could not connect to database template1: FATAL: role "<username>" does not exist
username is my username

I've seen some postgresql errors online talking about using the postgres user. Is this necessary?

Running on Ubuntu

node --version v13.3.0 npm --version 6.13.2

Thanks

Putting this into an issue as the last conversation about it seems to have been left open ended.

@janpaul123 recently posted this:

Apparently some potential Dynamicland collaborations have fallen through or are threatened by people setting up Paper Programs instead and calling it a day. That is the opposite of what I want to happen, especially since Dynamicland really needs funding.

It's easy to imagine how people could confuse Paper Programs for a OSS substitute of Dynamicland's interaction model: it superficially resembles Dynamicland, and the website doesn't do much to spell out the limitations of Paper Programs in comparsion, or if there is supposed to be any difference in direction.

Right now there is a disclaimer on the camera page:

Paper Programs would not exist without the incredible research community at Dynamicland. They continue to push the bounds of physical, communal computing, and continue to inspire. If you can, support Dynamicland by visiting, collaborating and donating. Thanks!! :)

This pushes people in the direction of Dynamicland, but still does nothing to suggest that there is any difference between the two, simply that Paper Programs has been inspired by Dynamicland.

Then there are issues in this repo that seem completely counter to any wish to reduce confusion between Paper Programs and Dynamicland, such as #30 ("Realtalk. Let's implement it"). Unless there is some collaboration with Dynamicland going on behind the scenes, this seems like exactly the opposite of what you'd want to do -- a kind of have your cake and eat it approach where it's stated that this shouldn't be used as a substitute for working with Dynamicland while actively working to duplicate it.

Finally, I consider myself fairly plugged in and follow this project closely -- but it's really not clear to me if this project aims to be anything else than a browser based clone of Realtalk.

What is the difference meant to be?

How is that difference expressed in the design?

What other things can be done to show joe-off-the-web that Paper Programs can't be used as a instant DIY Dynamicland kit?

Apologies for being a bit of a broken record, or if I'm stepping on anyone's toes. But it seems like the desire to play with Dynamicland's interaction model now at the expense of donations and collaborations with Dynamicland itself could be exactly the kind of short-termism that has made them hesitant to interact with the FOSS community.

I tried checking out paperprograms on windows10 and it seemed to work with a few tweaks except for an error related to pdfkit that is thrown once you visit localhost:3000:

Seems to be related to the PDFDocument-function under ./node_modules/pdfkit/js/document.js. I fixed it temporarily by replacing it with an empty function, but unfortunately I can't print any papers on windows for now. Can any other windows users having the same problem?