ykotseruba / pedestrianactionbenchmark Goto Github PK
View Code? Open in Web Editor NEWCode and models for the WACV 2021 paper "Benchmark for evaluating pedestrian action prediction"
License: MIT License
Code and models for the WACV 2021 paper "Benchmark for evaluating pedestrian action prediction"
License: MIT License
@ykotseruba I am geeting error when I am trying to test PCPA model. I did setup environment according to your docker file.
Please let me know if you have solution for this.
Thanks!
ValueError: Data is expected to be in format x
, (x,)
, (x, y)
, or (x, y, sample_weight)
, found: (array([[[[[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
...,
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612]],
[[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
...,
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612]],
[[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
...,
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612]],
...,
[[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
...,
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612]],
[[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
...,
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612]],
[[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
...,
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612],
[18.83785077, 19.85873724, 23.39030612]]],
[[[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
...,
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778]],
[[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
...,
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778]],
[[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
...,
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778]],
...,
[[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
...,
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778],
[20.46348852, 21.41812819, 25.08585778]],
Hi,
Your work is excellent and impressive. Establishing a benchmark for pedestrian action prediction is urgent because many papers use different methods of evaluation. But I have some questions about the benchmark mentioned in paper.
First, a pedestrian had the intention to cross but he(she) did not cross at last for some reason. Such samples are set to “Not-crossing”. But at that moment, this pedestrian really wanted to cross the road. Could you explain these samples?
Second, besides pedestrian action prediction, is pedestrian action detection also important? If a pedestrian is crossing the road, prediction is no meaning but action detection or recognition is necessary. So is necessary to add action detection or recognition in the benchmark?
Thanks!
Hi Yulia,
I notice that in pose_set01.pkl of JAAD, there are pose data for 294 videos. Are these train (177) + test (117)? I am wondering have you provided pose data for val subset? In the code, data_val can read pose data, but I am not sure why in this case there are only data for 294 videos in pose_set01.pkl.
Thanks!
First of all, I would like to congratulate you on this great work. I try to reproduce the results and see how to integrate a new approach into this benchmark. However, not having experience with Tensorflow, I'm having a little trouble debugging the error below.
In fact, I have a problem with Tensorflow learning part of the code. The generation of feautre is going well but the learning part is not working. Below is what I get as an error. I followed all the steps you recommended. I'm using Docker and I have version 2.16 of Tensorflow.
Thanks for your help.
python train_test.py -c config_files/PCPA.yaml
Total params: 31,165,953 (118.89 MB)
Trainable params: 31,165,953 (118.89 MB)
Non-trainable params: 0 (0.00 B)
### Class weights: negative 0.204 and positive 0.796 ###
lerreur commence par là je crois
Traceback (most recent call last):
File "/data/PedestrianActionBenchmark/PedestrianActionBenchmark/train_test.py", line 165, in <module>
run(config_file=config_file)
File "/data/PedestrianActionBenchmark/PedestrianActionBenchmark/train_test.py", line 107, in run
saved_files_path = method_class.train(beh_seq_train, beh_seq_val, **configs['train_opts'],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/PedestrianActionBenchmark/PedestrianActionBenchmark/action_predict.py", line 759, in train
history = train_model.fit(x=data_train['data'][0],
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/keras/src/utils/traceback_utils.py", line 123, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.11/dist-packages/tensorflow/python/data/ops/from_generator_op.py", line 124, in _from_generator
raise TypeError(f"`output_signature` must contain objects that are "
TypeError: `output_signature` must contain objects that are subclass of `tf.TypeSpec` but found <class 'list'> which is not.
Hi,
Thank you very much for your nice work!
Could you please tell what is the meaning of the 'overlap' in your paper and your code?
For example, in your paper you said 'The sample overlap is set to 0.6 for PIE and 0.8 for JAAD'. This is also set in the yaml file.
Many thanks again!
Bests,
Xingchen
Hello, Iuliia
I am a newly enrolled graduate student from China, and I am very interested in your research direction. You have done quite excellent work. Recently, I have been reading your paper "PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning." I am very eager to implement this work, but unfortunately, I was not successful. Could you please provide the source code for me? Thank you very much!
changsheng luo
Hi,
Thank you very much for your very intersting and excellent work!
I am wondering could you please provide some explanations about the pose data you provided, for example the data in the features/jaad/poses folder?
In the paper you mentioned you extract pose data using OpenPose, are the pose data (18 joints) included there? In addition, what is the meaning of 'pose_set01'?
Many thanks! Looking forward to your reply!
Bests,
Xingchen
Dear authors,
I am wondering if that is possible for you to provide your pretrained model of PCPA on the JAAD-beh dataset?
I have trained this model using the same parameters, however the results I obtained are much worse than what you reported in your paper. I got acc = 0.5, auc = 0.47, f1 = 0.59, precision = 0.6, recall = 0.58.
I wish to produce some qualitative comparison with PCPA, but it might be unfair (for example maybe I missed something so I do not obtain good performance) to use the PCPA results produced by myself.
Many thanks for your help!
Xingchen
Hi,
I have noticed that, in some examples, in the 16 past frames, the pedestrian is in the middle of the street and is already crossing.
Is it correct to have this type of behavior in the examples? I thought that the crossing behavior will happen only in the future frames.
For now, I've seen it in JAAD split. I share some frames.
Maybe I am doing something wrong but I am checking it right before the function train_model.fit() and I didn't modify the rest of the code.
Thank you very much for your time.
The configs in PCPA.yaml such as learning rate, epochs are different from that in the paper, especially for JAAD_behave, JADD_all datasets. Could you please update the configs for reproducing the results easily ?
Hi!
I used local context, speed, box, and pose in PCPA and trained the PCPA model with epoch = 80, LR = 5e-7. But I cannot get ACC = 0.85 in JAAD test dataset as described in the paper (I only got acc = 0. 79). Is there any detail I missed in the training process?
Thanks,
Haolin
hi,thanks for your work
When I reproduce the PCPA model, only set01,set02,set03,set04 are generated in the folder data/featrues/pie.Do you know what's causing it?
It seems that the dataset split is not the same as the paper and what you say in issue 8 https://github.com/ykotseruba/PedestrianActionBenchmark/issues/8#issue-823109452
Hi @PaulAdamastor,
Thank you for your interest in our work.
I'm not getting this error when I run the code of both models. Are you using the latest annotations and data loading functions (pie_data.py
and jaad_data.py
)?
To use I3D with the optical flow you can modify the C3D config as follows: change the model
to I3D
and obs_input_type
to [local_context_flow]
. Any type with visual features should work if you append _flow
to it.
However, the code expects the optical flow to be precomputed. We used FlowNet2.
Originally posted by @ykotseruba in #12 (comment)
Hello,Excuse me,
As shown in your paper, how do you extract a sample of pedestrians crossing or not crossing the street from the dataset and compare it with the results obtained from the model test? The test program is for the entire dataset, how to extract the test results for a certain pedestrian?
Hi,
I split PIE and JAAD dataset as your paper and codes. But I find imbalance of C/NC samples.
In PIE dataset, the numbers of different samples are as follows:
NC | C | |
---|---|---|
Train | 3576 | 1194 |
Test | 2742 | 1074 |
In JAAD_beh dataset, , the numbers of different samples are as follows:
NC | C | |
---|---|---|
Train | 374 | 1760 |
Test | 704 | 1177 |
In PIE dataset, the number of NC samples is far more than the number of C samples. In JAAD dataset, the number of C samples is more than the number of NC samples. I think it is harmful to train a model. Is the split result correct? Could you please explain this distribution?
Hi,
I am looking into your model and benchmark. Since you might be experienced in this topic, I'd like to ask about some details in C3D and PCPA model. From the aspect of coding, there are two operations: 1. directly using C3D model by calling C3D class; 2.keeping C3D model in PCPA class with removing other features: speed, box, pose as well as the attention modules. Will these two operations get same results? In my view, they both use only C3D model for prediction. Otherwise, do they have differences in detail?
I am interested in these models and want to do some exploration based on your repo. Could you answer it for me?
Thanks,
Haolin
Hello,
Would it be possible to put a zip file with all the pretrained models evaluated on the paper?
Thank you very much :)
Hi! Yulia,
I met the problem in the environment configuration. Could you tell me the exact version of CUDA and Cudnn you use in your PCPA model's training process? I used TF2.1+CUDAtoolkit10.1+Cudnn7.6.5 but meet the unknown problem of initialization of Cudnn. Btw, I used Anaconda rather than Docker. Is there anything I need to take care of?
Thanks,
Haolin
Hello,
thank you for the nice code and benchmark!
I've tried running a few training but I have issues with some models:
ATGC and TwoStream both give me the error:
"'walking': data_raw['actions'].copy(),
KeyError: 'actions'"
Could you help me with that?
And I didn't really understand how to use optical flow with I3D, could you provide me a config file?
Best regards,
Hi,
As you mentioned in https://github.com/ykotseruba/PedestrianActionBenchmark/issues/3#issuecomment-782693588, the number of samples in JAAD decreased. Could you please provide the updated evaluation results in paper Table 2, 3?
Thanks
Hi,
After I retrained PCPA with the settings in the paper, the performance was not as good as the data in the paper. While you were answering your other question, I noticed that the performance on JAAD would drop due to some issues, but it was still higher than my results. I use the windows system to train and do not use docker. I want to know if this is the main reason and whether using docker will affect the experimental results.
Thanks for your answer.
It seems that these two file from the original repo cannot run the code successfully, I think the jaad_data.py and pie_data.py may be modified.
Hello,
I have a question regarding the usage of the DataGenerator in the PCPA Modell.
When I use the DataGenerator I get very good results, but a high loss (0,13). When DataGenerator = False the results are very bad, but the loss is very low (0,003).
Shouldn't the results be almost identical? What could be the reason for this?
Many greetings
Moritz
In your docker file, it shows
FROM tensorflow/tensorflow:latest-gpu
Hi,
In your paper, I find this sentence:
"In the PIE dataset, we follow the data split defined in [42]: videos from set01, set02 and set06 are used for training, set04 and set05 for validation and set03 for testing. The number of pedestrian tracks in PIE is 880, 243 and 719 in train, validation and test sets."
After running codes, the PIE dataset split is different from the paper. Videos from set01, set02 and set04 are used for training, set05 and set06 for validation and set03 for testing. But the number of pedestrian is consistent. Finally, I find these codes in pie_data.py.
def _get_image_set_ids(self, image_set):
"""
Returns default image set ids
:param image_set: Image set split
:return: Set ids of the image set
"""
image_set_nums = {'train': ['set01', 'set02', 'set04'],
'val': ['set05', 'set06'],
'test': ['set03'],
'all': ['set01', 'set02', 'set03',
'set04', 'set05', 'set06']}
return image_set_nums[image_set]
Do I understand correctly? Could you tell me which PIE dataset split is correct?
Thanks a lot!
command: python3 train_test.py -c ./config_files/C3D.yaml
error: https://i.imgur.com/hkFfpJS.png
running C3D crashes when downloading the model in line https://github.com/ykotseruba/PedestrianActionBenchmark/blob/main/action_predict.py#L1178 because the "weights" folder does not exist.
a possible solution would be os.makedirs(os.path.dirname(self._weights), exist_ok=True)
before that line.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.