@ZikangZhou Hi，论文里面说“During online running, the perception module supplies a stream of agent states to the prediction
module at a fixed interval, where each agent state is associated with its spatial-temporal position and geometric attributes”，但是在源码里面并没有找到“the perception module”，请问“a stream of agent states ”是直接由数据集Argoverse 2提供的吗？

The paper states:

Benefiting from modeling in local reference frames, the embedding of each agent state/map polygon has only one instance and can be reused in the subsequent observation windows.

As the Argoverse 2 dataset contains independent scenarios as samples, the scene features are encoded from scratch for every sample and it is not possible to reuse any embeddings. If I want to apply your model on a real life use case where the next sample is the next observation window in time, how would I implement this reusing of the agent state/ map polygon encodings?

Thank you for your wonderful work. We tried to implement QCNeXt based on the report :https://arxiv.org/abs/2306.10508 . However, I could not manage to reproduce the reported metrics. We made the adaptation based on this repo and tried to align every detail as described in the technical report.

@ZikangZhou Hello，thanks for your excellent work. I would like to ask a question, when configuring the environment, in step 3 of "Getting Started", is this step directly installed in the conda environment created in step 2 of "Getting Started", or does it need to create a new conda environment ? Because step 1 in installing the Argoverse 2 API is "bash conda/install.sh && conda activate av2", which will create and activate a new conda environment?

Thank you for your wonderful work and code sharing, I would like to ask that when I train from scratch my results are two points worse than yours on minFDE on both the evaluation and validation sets, my training parameters are the same as yours, any tips on the training process please, looking forward to your answer!

请问有相关轨迹预测可视化的实现代码吗？

hellow! I wonder what "pt" and "pl" stand for in code respectively? is "point" and "polygon"?

周博士：
您好！

我在阅读您的代码时注意到了一些关于加快推理速度的技巧，特别是关于cache的使用以及如何处理帧与帧之间的agent编码问题。我有几个问题想要进一步了解和确认：

• Temporal Attention中的KV Cache使用：在处理新帧时，您使用了历史的agent encodings。我理解这里应该使用的是KV Cache技巧，通过下三角矩阵的mask来保证推理结果的一致性。请问我的理解是否正确？

• Agent数量与顺序不一致的处理：

  • 在连续的帧中，agent的数量可能不同，例如上一帧有5个agent，新一帧只有2个。如果past的维度是A, T-1, D，比如5, T-1, D，而新一帧的维度是2, 1, D，这种情况下应该如何合并这些数据？
  • 如果agent的传入顺序在不同帧中不同，比如上一帧是X, Y, Z，新一帧是Y, Z, X，这在合并时可能会造成问题。应该如何处理这种顺序变化？

• 单帧Agent的edge和r计算：当只传入一个时间点的agent数据时，例如计算edge_index_t可能为空，这会导致无法计算rel_pos_t等相对关系，进而影响后续的Temporal Attention计算。我试了试每个agent的edge_index_t给的值为[[0],[0]]，但attention的结果略有差异。这种差异可能是因为单个时间步的position_embedding和连续时间步骤的position_embedding不同导致的，在这种情况下应该如何处理能保证推理和训练结果一致性呢？

附上相关我理解的代码修改段落供参考，麻烦看看我的理解是否正确：

for i in range(self.num_layers):
    x_a = x_a.reshape(-1, self.hidden_dim)
    x_a = self.t_attn_layers[i]((x_a, x_a), r_t, edge_index_t, kv_cache = kv_cache)
    x_a = x_a.reshape(-1, 1, self.hidden_dim).transpose(0, 1).reshape(-1, self.hidden_dim)
    x_pl = x_pl.transpose(0, 1).reshape(-1, self.hidden_dim)
    x_a = self.pl2a_attn_layers[i]((x_pl, x_a), r_pl2a, edge_index_pl2a)
    x_a = self.a2a_attn_layers[i]((x_a, x_a), r_a2a, edge_index_a2a)
    x_a = x_a.reshape(1, -1, self.hidden_dim).transpose(0, 1)

if x_a_past is not None:
    x_a = torch.cat([x_a_past[1:, :, :], x_a], dim = 1) 
return x_a, kv_cache

期待您的回复和指导。

谢谢！

Hi, Zikang. Thanks for your nice work. May I ask have you tried the model on the waymo dataset? If there is a code for testing on the waymo dataset, could you share it with us? Thanks a lot!!!

I am really impressed by the work. However, reading through QCNet paper and QCNeXt report, you have mentioned that the encoders used are the same. Whereas the Fig 1 of QCNeXt technical report shows an encoder that is different from the QCNet encoder. Also, the encoder shown in the QCNeXt, is kind of what is shown in the QCNet video presentation for the existing works that are computationally expensive. Can you please clarify for better understanding? Also any expected time frame for release of QCNeXt code?

I wonder how to load checkpoint and continue training. Because of the OOM (maybe at 30 epochs), I have to load checkpoint and continue training after OOM. I try to add ckpt_path = '....ckpt' (path from lightning_logs/checkpoint), but it seems useless. How should I solve this problem？ Thanks a lot!

I read your paper and I have some questions about the multi-agent forecasting. I would like to apply your model to a real life use-case and forecast trajectories for multiple agents.
Is this already possible with the code you provide? To my understanding, the decoder needs to know which target agent it should predict trajectories for. Can I just implement multiple decoder heads with shared encodings and how would I tell the decoder which agent it should predict trajectories for?

Thanks in advance!

Hello，Dr Zhou，请问，1. 若想在现实中实现这个模型或HiVT模型，必须像argoverse数据集那样给出所有agent在空间中的绝对坐标吗？
2. 现有的车辆轨迹预测方法大多数都是在公开数据集中进行的，若想实现这些方法，是否也需要提前将环境信息（车道线，交通灯，人行道等）构建好？
期待您的解答，谢谢！！！

hello.

I proceeded from QCNet train to validation for path prediction.

Could you please provide the ipynb file and code for visual testing such as Qualitative Results?

And, I am trying to validate a model already trained with num_future_steps =60, num_historical_steps=50.
Can you please suggest the code to get minADE, minFDE for num_future_steps 0 to 10 or 0 to 20 in validation?

Hi, when I conduct conda env create -f environment.yml, there will be some version conflicts of conda.
4.5.4 and 4.12.0 both have problems, but 4.10.1 can succeed.
Can I know your conda version so that I can reproduce your great work in omy own computer.

Hi, your work is great. I try to generate the prediction results on the test set by following your code:

python test.py --model QCNet --root /path/to/dataset_root/ --ckpt_path /path/to/your_checkpoint.ckpt

But there is some error like this:

Global seed set to 2023
Processing...
  0%|          | 0/24984 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "test.py", line 40, in <module>
    test_dataset = {
  File "/content/QCNet/datasets/argoverse_v2_dataset.py", line 150, in __init__
    super(ArgoverseV2Dataset, self).__init__(root=root, transform=transform, pre_transform=None, pre_filter=None)
  File "/usr/local/envs/QCNet/lib/python3.8/site-packages/torch_geometric/data/dataset.py", line 97, in __init__
    self._process()
  File "/content/QCNet/datasets/argoverse_v2_dataset.py", line 534, in _process
    self.process()
  File "/content/QCNet/datasets/argoverse_v2_dataset.py", line 190, in process
    map_data = read_json_file(map_path)
TypeError: object() takes no arguments

Wondering what's the reason and how to deal with it. Thank you very much.

@ZikangZhou Hi, thanks for your awesome work. Would the multi-world version be also released in this repo?

@ZikangZhou Hi, I followed the script provided in your readme file to reproduce the results, but encountered two issues.
Firstly, when I set the batch size to 4 on my 3090 GPU, I encountered out of memory problem.
So I adjusted the batch size to 2, while keeping the other hyperparameters unchanged.
The resulting reproduction shows that minade=0.732 and minfde=1.294, which has a small deviation from the results reported in the paper.
However, there is a significant difference from the pre-trained model that you uploaded, where minade=1.25.
I would like to ask if the hyperparameters or other training configurations of the pre-trained model in this version have been changed?
Thank you very much.

您好，请问这个是在ubuntu系统下运行的吗，在windows下可以运行吗

Hi I have been training the model for two days now on 2 x V100 GPUs 32 GB. This is what the output at my terminal looks like. Can you comment on if the training is going in the right direction? I see some losses increasing here. Also how many epochs is the training suppose to last for?

@ZikangZhou Hi Zhou! I noticed that when I train this code, it utilizes 32GB of GPU memory per GPU instead of the 20GB mentioned in the README.md. Could you please explain what might be causing this discrepancy in this repository?

Hi, can you share the Inference time (ms) and GFLOPs of QCNet and QCNeXT projects. Something very similar to this paper that reported these metrics for other frameworks in the paper titled :

ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposals

I have already prepared the dataset allow the correct instructions, but i met the problem below, how can i fix it?

The Brier score in my training is not decreasing while all the losses and the socre seems to be converging to the reported values. The corresponding value after 68 hours of training on A100, with 100GB RAM.

The training hyperparameters are:
--num_workers=32 --train_batch_size=16 --val_batch_size=16 --test_batch_size=16 --max_epoch=50 --pl2pl_radius=150 --time_span=10 --pl2a_radius=50 --a2a_radius=50 --num_t2m_steps=30 --pl2m_radius=150 --a2m_radius=150

Current training status;
Epoch: 32
Training duration: 68 hours

Brier: 0.61 (osciallting near this value since the begining of the training)
MR (k=6): 0.1824
minADE (k=6): 0.751
minFDE (k=6): 1.359

I am also attaching the Tensorboard checkpoint for further analysis. Please let me know if anything else is required.
Checkpoint.zip

This is more a question than an issue.

Was there an interest to try the PyG TemporalDataLoader to increase performance?
I believe there is some information that can be obtained from feeding the temporally linear data together.

I don't know much of Argoverse2, but I know Argoverse1 sequences were not temporally linear so there was no possibility to do this in HiVT.

hello I want to know when will update qcnext

The paper is claimed to be capable to predict agent trajectories without hd-maps data.
Could you provide some demo on how the model predicts trajectories in such scenario?

Hi Zikang,

Cheers for your wonderful work! Is it possible for you guys to release the visualization code as well? As I'm not quite familiar with the argoverse dataset and its toolkit, it would be great if you could share your visualization methods. Thanks!

Thank you for your amazing work and contribution to the trajectory prediction community, since the model is big and requires large resources to train, how long did you train the model with those resources?

How to solve this issues? I am getting this error during training.

av2-api有安装教程嘛？我之前使用过程中有找不到upath的问题。from upath import UPath找不到这个UPath，网上也没找到对应解决方案。

av2-api有安装教程嘛？我之前使用过程中有找不到upath的问题。from upath import UPath找不到这个UPath，网上也没找到对应解决方案。

Hi,Doctor Zhou:
Well done work!But I have some problems.
1.What causes the OOM problem during training?Recently,someone says that Argoverse2 dataset has different size of map representation input.When training,random combination of large size of samples in a batch may leads to OOM problem.So how can I deal with the problem with a limited memory of GPU.
2.In one of QCNet improvement,after transpose and rotation invariance processing,does each agent in each timestep has a local reference and each agent in each timestep can be regarded as a node in GAT?The edge is the relative position relations of two agents' local reference?I noticed that the relative position relations of two agents' local reference is represented by the relative location,relative direction,relative orientation and relative time-order.I wonder if the four items can uniquely determined the relative relations of two local references?

周博士您好，
想咨询下，您是否有计划公开argoverse1数据集处理的代码？
祝好~

在qcnext中，第二阶段的的anchor pos embedding是如何生成的

Hello, author. I did not find the "anchor-free mode queries" mentioned in the relevant paper in the code. Excuse me, where do you implement "anchor-free mode queries" in your code?