Comments (4)
I'm not sure what your output corresponds to, I'm sorry. My best guess is that you are writing a datalog-alike, and using distinct
, which has the property that it maps all non-zero occurrences to one occurrence. If that isn't what you are seeing, feel welcome to provide some more detail! Good luck.
from differential-dataflow.
Thanks for the fast reply @frankmcsherry .
To implement the rule I've stated I have used:
let mut r_in = InputSession::new();
let probe = worker.dataflow(|transitive_closure| {
let r = r_in.to_collection(transitive_closure);
r
.iterate(|inner| {
let join = inner
.map(|(x,y)| (y,x))
.join(&inner)
.map(|(_, (x,z))| (x,z));
inner
.concat(&join)
.consolidate()
.distinct()
})
.inspect(|x| println!("r{:?}", x))
.probe()
});
I make use of distinct()
to avoid duplicate tuples. It is still not clear to me why is this affecting the removal behaviour, could you please elaborate more?
from differential-dataflow.
from differential-dataflow.
Ok, I understand the effect of distinct() now. However, in the following program (reads in inputs like +1,a,b or -1,b,c) I do not use the distinct operator, and still see that removing two tuples which do represent a join match, actually do yield an update with multiplicity "1" ...which looks a bit odd, but is due to the multiset logic/semantic?
fn main() {
timely::execute_from_args(std::env::args(), |worker| {
let mut input = InputSession::new();
let probe = worker.dataflow(|transitive_closure| {
let input_collection = input.to_collection(transitive_closure);
input_collection
.inspect(|tuple| println!("in: {:?}", tuple))
.map(|(x,y)| (y,x))
.inspect(|tuple| println!("flip in: {:?}", tuple))
.join(&input_collection)
.inspect(|tuple| println!("join match: {:?}", tuple)).probe()
});
let mut time = 0;
input.advance_to(time);
loop {
let mut line = String::new();
let _ = io::stdin().read_line(&mut line);
let args: Vec<&str> = line.trim().split(",").collect();
match args[0].to_string().trim() {
"+1" | "1" => input.insert((args[1].to_string().clone(), args[2].to_string().clone())),
"-1" => input.remove((args[1].to_string().clone(), args[2].to_string().clone())),
_ => unimplemented!(),
};
time = time+1;
input.advance_to(time);
input.flush();
while probe.less_than(&input.time()) { worker.step(); }
}
}).unwrap();
}
from differential-dataflow.
Related Issues (20)
- Strategies for maintaining persistent states (the data in Collections) HOT 2
- Consolidate Timestamps and Time Windowed Dataflows
- what different with flink Retraction
- Optional Abomonation? HOT 1
- Replicate Cross Join Situation HOT 2
- Operator to flatten `Collection<Collection<G, D, R>>` into `Collection<G, D, R>`
- Difficulty understanding how to use prefix_sum / how to implement topK HOT 6
- miri: Undefined Behavior: trying to retag from <20432167> for Unique permission in push_unchecked HOT 2
- Support `TimelyStack` as storage for `(T, R)` in arrangement leafs HOT 1
- Holding on to a trace with physical/logical compaction to the empty frontier stalls compaction
- Question: how to change data timestamp for late arriving data HOT 4
- Question: how to query data from past timestamps? HOT 5
- maybe the doc should add some instructions at geting started section
- Does all data have to be in memory? HOT 3
- Revisit the stashing logic in MergeBatcherColumnation
- Arrangement batch formation costs in proportion to outstanding updates HOT 3
- Suggestion: WebAssembly support HOT 5
- Getting Started Guide for Newcomers Doesn't Work HOT 3
- Improve clarity around `Cursor` method requirements
- Implement flat container support for `PointStamp`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from differential-dataflow.