Hello,

Situation:
Person A is tracked and is about to leave the window of observation
Person B is not tracked yet and will cross the way of Person A (at the observation's border).

I oberseved that the tracker is often wrong and will say that Person A is Person B (like if Person A is going backward).

Is there a way to avoid this kind of error? I thought to code an algrotihm that avoid that the tracker goes backward but if you have any trick it could be cool!

Thanks in advance ;)

I tried to run this file to generate features, after the first frame are processed, the program is killed due to this following error:

InvalidArgumentError (see above for traceback): NodeDef mentions attr 'identical_element_shapes' not in Op<name=TensorArrayV3; signature=size:int32 -> handle:resource, flow:float; attr=dtype:type; attr=element_shape:shape,default=; attr=dynamic_size:bool,default=false; attr=clear_after_read:bool,default=true; attr=tensor_array_name:string,default=""; is_stateful=true>; NodeDef: net/map/TensorArray = TensorArrayV3clear_after_read=true, dtype=DT_FLOAT, dynamic_size=false, element_shape=, identical_element_shapes=true, tensor_array_name="", _device="/job:localhost/replica:0/task:0/gpu:0". (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
[[Node: net/map/TensorArray = TensorArrayV3clear_after_read=true, dtype=DT_FLOAT, dynamic_size=false, element_shape=, identical_element_shapes=true, tensor_array_name="", _device="/job:localhost/replica:0/task:0/gpu:0"]]

Would you kindly show me how this can be fixed? Thank you!

Best regards

Hi nwojke，

I find that after the maxpool the feature map size becomes 63x31 instead of 64x32. I guess after conv2d with stride 2 the size becomes 32x16 as expected. I wonder is it correct running in the case of abnormal size?

Hi nwojke,

I ran into issues where model can not track object in low frame rate videos ( ~ 5 fps). It works fine when I test on higher frame rate videos (~15 - ~25 fps).

For example, object 1 at frame i is at top left. At frame i + 1, the object moves to the middle. I wonder if there is any parameter I should consider updating (in Kalman filter?) to accommodate this problem?

Thanks!

P/s: great work!

Thanks for your wonderful code ~But when I run the code on the 7 train sequences from MOT16 ,I got an MOTA of 17.9.%.The detail of result is as followed. But on the mot16 results page, the deep_sort get a MOTA of 35.3%. I donnot know the so big difference between each other. The detection we used is the public detection. I try to upload the result on the test sequences of MOT16 in the server,but I have not made it.Thanks for your reply.
********************* Your Benchmark Results (2D) ***********************
IDF1 IDP IDR| Rcll Prcn FAR| GT MT PT ML| FP FN IDs FM| MOTA MOTP MOTAL
17.8 32.2 12.3| 43.9 63.4 5.26| 517 65 232 220| 27964 61901 729 1680| 17.9 76.9 18.6

From what I see and read, I have to train the model etc. Would this method work on realtime videos?

For example I analyze the frame detect the BBs and pass it on directly to the feature network. Can it work like that?

Hi,
I want to run the algorithm on ROS.
I have built the methods that I will need in getting images and their corresponding detections at the same time in the correct format.
I am not sure where I should change the deep_sort_app.py to take input from detections and image messages from ROS. Any insights?

Thanks!

Hi,

I wanted to ask that how can I use deepSort to track street lights. What model do I need to change? And can I retrain this model to use for detection of street lights?

Your help would be really appreciated. Thanks

The paths are correct and directory structure is as mentioned.
Tracking demo works fine, problem arises at feature generation, running the command mentioned in your README. TF version is 1.3, mars-small128.ckpt is there.

Any ideas? Thank you!

OSError: [Errno 2] No such file or directory: './MOT16/test/MOT16-06/img1'

I have multiple videos and there corresponding detection files . So i want to store track state of first video in file and use this track state detection file for next video.

When I run deep_sort_app.py with --display=False or --display=false, the tracking result is still displayed, after searching on google, it seems that argument with bool type in argparse doesn't work, maybe it's better to delete the type=bool and add action='store_true' for argument display.

Hi, Thanks for you work!
the paper(SIMPLE ONLINE AND REALTIME TRACKING WITH A DEEP ASSOCIATION METRIC) mentioned a method that use Mahalanobis distance to get the motion information, however, I only saw the IOU method, so did the project use Mahalanobis distance method? if used, which file include it?
Thanks a lot

Hi, nwojke, although it is said that it is a bit hacky to set the motion and observation uncertainty, it seems that there is a pattern in kalman_filter.py when you initiate covariance in initiate phase, compute motion_cov in predict phase and innovation_cov in project phase, can you show me some reference or suggestion to set these variables? Thank you.

Thanks for the amazing work and open source!
Could you help me with the following questions please:

1. Why default --nms_max_overlap=1? It means no suppression will be performed right?
2. Was it experimentally best threshold for MOT16?

Thanks.

line 172 , detections = [d for d in detections if d.confidence >= min_confidence]

I want to print the line 172 detections in deep_sort_app.py
but it show the
[<deep_sort.deep_sort.detection.Detection object at 0x7f87fe29c710>]

have any method to print value?

Can you point me in a direction to track pedestrians in realtime basis?

Hi,

I wanted to know if the deep sort can be used to track objects from RADAR data. I have used the sort for images and it works great. But will it work for RADAR+Camera fusion or just radar detections?

Thanks!

HI
if when t = 1, the deep sort detect the person A and save the feature of the person A ,
when t = 2, the deep sort detect unknown person ,it will use cnn to match the feature of the person A at t=1 and the feature of unknown person at t=2, if the result is ture (same person ) , the deep sort will save the feature of the person A at t=2 , and the feature of the person A at t=1 will forget , right??

thanks

There are many detection methods like Faster RCNN, Yolo, etc. So, which method is best suited for tracking. Currently I am using YOLO V2 but tracking results are not that much good. Occlusion is still there. Can you give me any suggestions to improve performance of tracking.

seems that hit_streak variable in track.py is a dummy variable, is not used at all.

Hi, thanks for the wonderful work. I have only one question, I print out the detections.npy file, the shape
of MOT16_POI_train/0004 is ( 40709, 138), 138 is the last dim of featuremap with no doubt.
But 40709 don't make sense, because MOT16_POI_train/0004 has 1050 images.
if every image has a single 1-d featuremap which should be (1050, 138), So basically, what is the shape of extracted feature map?

in tracker.py line 127
unmatched_tracks = list(set(unmatched_tracks_a + unmatched_tracks_b))

how about the situation that the track included in both unmatched_tracks_a and iou_track_candidates, but excluded in unmatched_tracks_b? as this track should be matched.

Using deep SORT in your own project?
how to do it ?
thanks

Just want to know if the CNN is trained based on a weight-sharing Siamese network with triplet loss head? Or is it trained with a softmax loss where the number of classes equals to number of person IDs?

I am wondering how to train with different object other than pedestrains. Is there an issue time for training code? Thanks in advance.

Hi Nicolai,

I am working on an MOT project for tracking several different object classes (cars, pedestrians, cyclists, etc.). I understand that the model provided in your github repository for Deep SORT is trained on the MARS dataset which is specific for person re-identification, and was wondering if you could provide the code for training your CNN architecture so that I can train it on my own dataset.

Alternatively, it would be good if you could share some pre-trained models for object classes other than pedestrians.

Thanks for your work on Deep SORT. Cheers!

the ci,j = λ d(1)(i, j) + (1 − λ)d(2)(i, j) is in the paper (5)

where is the code to do the (5)?

hi,
any upcoming training - testing new tracking apps, trained from scratch examples(non pedestrian) predicted soon?

Hi @nwojke,

What is the code's license ?
Thanks

Hello,
I am unable to download pre-generated detections and the CNN checkpoint file from the network drive. I am keep getting " File is currently busy, please try again later".

I am just checking if the server is busy, or the files are removed. I can see the file, my download by clicking a file does not work.

Thanks,
Fayzur

I want to know is there any way to optimize the time by running generate_detections side by side while we are running object detector to get the B-Boxes... I referred the issue #4 I want the flow like:

1. Detect_objects in frame -> after detecting objects run generate_detections to get encodings for .npy file -> after video gets finished store all the encodings in .npy file -> run deep_sot_app

Is there any code showing the loss function to train the current residual network?
What the authors think about having a single network to do detection and feature extraction? Is it possible?

I want to make a .pb file for your model. However I couldn't identify the output node name. Any suggestion?

Hi, could I get MOT16 labels for detection which can be used for training a detector?

Hello. I want to track people using on my dataset, but I have a problem about adapting it to the project. In this case, do I have to provide .det file?

Hello，
When I use the deep sort in py-faster-rcnn, there are some errors bescause the py-faster-rcnn is caffe,the deep sort is tensorflow, so I change the generate detection code, but when use the generate_detections.create_box_encoder to get encoder, I get the error: RuntimeError: Attempted to use a closed Session. I don't know how to solve this problem!
thanks~

Hello, I've managed to run the tracker correctly with video as input, and I'm using YOLOv2 for generating box detections,but I got really bad results for the tracking, here is the snipet of the code I'm using :

metric = nn_matching.NearestNeighborDistanceMetric(
    "cosine", 0.2, 100)
tracker = Tracker(metric)
encoder = generate_detections.create_box_encoder(
        "resources/networks/mars-small128.ckpt-68577")

camera = cv2.VideoCapture(file)
while camera.isOpened():
        _, frame = camera.read()
        if frame is None:
            print ('\nEnd of Video')
            break
      h, w, _ = frame.shape
	thick = int((h + w) // 300)
##################Yolo part to generate detections #####
        detections = []
        scores = []
        boxes = self.sess.run(self.out, feed_dict)
        for b in boxes:
                left, right, top, bot, mess, max_indx, confidence = boxResults(b)
                detections.append(np.array([left,top,right-left,bot-top]))
                scores.append(float('%.2f' % confidence))
        detections = np.array(detections)
##################################################
        features = encoder(frame, detections)
        detections = [Detection(bbox, score, feature) for bbox,score, feature in
                        zip(detections,scores, features)]

         # Run non-maxima suppression.
         boxes = np.array([d.tlwh for d in detections])
         scores = np.array([d.confidence for d in detections])
         indices = prep.non_max_suppression(boxes, nms_max_overlap, scores)
         detections = [detections[i] for i in indices]

         tracker.predict()
         tracker.update(detections)

         for track,det in zip(tracker.tracks,detections):

	          bbox = track.to_tlbr()
		  cv2.rectangle(frame, (int(bbox[0]), int(bbox[1])), (int(bbox[2]), int(bbox[3])),
					        (255,255,255), thick)
		  #cv2.putText(frame, str(track.track_id),
                                         #(int(bbox[0]), int(bbox[1]) - 12),0, 1e-3 * h, (255,255,255),thick//3)

		  bbox = det.to_tlbr()
		  cv2.rectangle(frame,(int(bbox[0]), int(bbox[1])), (int(bbox[2]), int(bbox[3])),
					        (255,0,0), thick)
	          cv2.putText(frame, str(track.track_id),
                                                (int(bbox[0]), int(bbox[1]) - 12),0, 1e-3 * h, (255,0,0),thick//3)
        cv2.imshow('', frame)

Here is the the results I got after testing, the ID's were not stable , and the white boxes (generated by the tracker) were always thin (width =0) :
video in google drive (in blue are yolo detections , and white output of the tracker )
can someone please help find where is the problem , and a possible solution ?
thank you !

Hi !

Do you know if there is a simple way to run Deep Sort using detection file + video file using open cv instead of using a folder of images?

Thanks!

Hi ,

Thanks for your code.
I tried to make a better understand your code with some source of kalman filter.
On the line of 184 of kalman_filter.py:

184 new_covariance = covariance - np.linalg.multi_dot((
kalman_gain, projected_cov, kalman_gain.T))

I can't understand it well.Because it seems conflict with my resource of kalman filter.

Can you make some explanation ? thx

hello, I have a problem that the track_id add 1 each frame. detection with faster rcnn, and tracking with deep sort, but the results are error, can you tell me how to solve this problem?

How can this code be used for "real time" video sequence from a web camera? As far as I see from the instruction, in order to track people of a video sequence, a npy file, which is generated in advanced, needs be to provided. and the npy file only works for designated video sequence, so it is not general. please correct me if I am wrong, and it is greatly appreciated if any hint is given on how to adapt the code for real time web camera video sequence. thx

Hello, I train car model with triplets Re-id, and I got for files that .data, .index, .meta. The deep sort use a .ckpt file for model, but I train model no have this file, so how can I do? Do you give me some suggest?than you!

As the title. I want to transplant the project to C++. What is the probability?

hello,
I want to use deep_sort in my own videos , but I don't have "det" file. How can generate "det" file like MOT 16?