Comments (2)
Hi, your understanding of the loss is correct. While it makes a lot of sense, we didn't use it because we found it doesn't improve the performance much. My intuition is that if we have a pair of cycle-consistent 2D correspondences A and B, by enforcing A maps to B and B maps to A, the depth loss will be somewhat automatically fulfilled. Because, let's assume that A already maps to B (with very peaky opacity, i.e., the first nonzero sample on A's ray has opacity=1 and flows to a 3D location XB that projects into B's location), then in order for B to map back to A, a sensible solution is just to make all samples whose depth is smaller than XB to be zero, so that B's corresponding 3D location will be XB which will be mapped back to A by the invertible network. So by enforcing the correspondences to be cycle consistent + the invertible network, the depth consistency is enforced as well.
Yes, adding depth supervision is possible, and we tried something similar too. But I'm not sure if this would improve tracking performance. But it may be that we didn't try hard enough. It seemed to me that by simply adding a scale/shift invariant depth supervision, there is tension between satisfying the flow loss vs the depth loss, which makes the optimization harder. One can also make major changes, e.g., adding depth supervision but also adding camera poses and using perspective projection (so that the static part can now really have zero deformation which will make the job of the invertible network easier), but this can be a bit too complicated. Exploring how to add depth to allow better handling of occlusion is definitely worth exploring
from omnimotion.
Thank you so much for your detailed response. I'm really grateful for it.
It seemed to me that by simply adding a scale/shift invariant depth supervision, there is tension between satisfying the flow loss vs the depth loss, which makes the optimization harder.
In fact, I already tried this simple way of adding scale/shift invariant depth supervision. The results prove the tension between flow and depth loss you mentioned. They cannot be optimized together in this way from my experiments.
One can also make major changes, e.g., adding depth supervision but also adding camera poses and using perspective projection
It also came to me the idea of adding camera pose and using perspective projection to supervise on static areas. I'll try this one though I haven't got any idea about how to take occlusion into consideration in this supervision now. I'll think about it carefully.
Thanks again and I'll appreciate it if we can keep in contact using e-mail when it's convenient for you.
from omnimotion.
Related Issues (20)
- preparation of custom data set
- Training and evaluating model on TAP-Vid DAVIS produces different results HOT 10
- How to accelerate training speed? HOT 1
- Reporting mistakes during training HOT 2
- A question about blending weight. HOT 9
- The TAPNet loader Module
- The given checkpoint do not match all the model, and it's hard to reproduce the result HOT 6
- Hello! This is a question about how to perform online operations after training is complete. HOT 3
- Train all frames or sample some? HOT 2
- Particle Tracking Results HOT 3
- Does it have to be trained and optimized for every new video? HOT 1
- the frame resolution when evaluating on TAP-Vid HOT 1
- Will the model weights and testing code be open-sourced HOT 1
- Transformation matrix
- Evaluate the trained checkpoints and the provided checkpoints, and the results of the metrics are inconsistent
- What may be the reason for not generating visual trajectories.
- Can I transform the input of INN into three-dimensional coordinates with an additional fixed fourth dimension?
- track fast moving objects
- Could you share the code for RAFT-C and RAFT-D evaluation in table 1? Thank you! HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from omnimotion.