diff --git a/README.md b/README.md
index 4de982d..9e9bd25 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,9 @@
## IOU Tracker
-Python implementation of the IOU Tracker described in the AVSS 2017 paper
-[High-Speed Tracking-by-Detection Without Using Image Information](http://elvera.nue.tu-berlin.de/files/1517Bochinski2017.pdf).
+Python implementation of the IOU/V-IOU Tracker described in the AVSS 2017/2018 papers:
+
+[High-Speed Tracking-by-Detection Without Using Image Information](http://elvera.nue.tu-berlin.de/files/1517Bochinski2017.pdf)
+
+[Extending IOU Based Multi-Object Tracking by Visual Information](http://elvera.nue.tu-berlin.de/files/1547Bochinski2018.pdf)
This project is released under the MIT License (details in LICENSE file).
If you think our work is useful in your research, please consider citing:
@@ -15,27 +18,95 @@ If you think our work is useful in your research, please consider citing:
ADDRESS = {Lecce, Italy},
URL = {http://elvera.nue.tu-berlin.de/files/1517Bochinski2017.pdf},
}
+
+@INPROCEEDINGS{1547Bochinski2018,
+ AUTHOR = {Erik Bochinski and Tobias Senst and Thomas Sikora},
+ TITLE = {Extending IOU Based Multi-Object Tracking by Visual Information},
+ BOOKTITLE = {IEEE International Conference on Advanced Video and Signals-based Surveillance},
+ YEAR = {2018},
+ MONTH = nov,
+ PAGES = {441--446},
+ ADDRESS = {Auckland, New Zealand},
+ URL = {http://elvera.nue.tu-berlin.de/files/1547Bochinski2018.pdf}
+}
+```
+### Table Of Contents
+- [Install](#Install)
+- [Demo](#Demo)
+- [DETRAC](#DETRAC)
+- [Motchallenge](#Motchallenge)
+ * [MOT16](#MOT16)
+ * [MOT17](#MOT17)
+ * [CVPR19](#CVPR19)
+- [Visdrone-MOT](#Visdrone-MOT)
+- [Contact](#Contact)
+
+**Update** (Jan 2020):
+* added V-IOU Code
+* updated README.md
+
+**Update** (December 2018):
+* added V-IOU results of our new paper [Extending IOU Based Multi-Object Tracking by Visual Information](http://elvera.nue.tu-berlin.de/files/1547Bochinski2018.pdf)
+* Mask R-CNN detections for UA-DETRAC added
+* CompACT parameters improved
+
+### Install
+The repository now contains the code for both the IOU tracker and the V-IOU tracker.
+The IOU Tracker only depends on numpy while V-IOU also requires OpenCV-Contrib and some other dependencies.
+It is recommended to use a virtual environment to run the code:
```
+virtualenv -p python3 env
+source env/bin/activate
+pip install numpy lapsolver tqdm opencv-contrib-python
+```
+This should get you started with a basic installation to run most the scripts in this repository.
+
+#### KCF/KCF2
+Two different implementations of the KCF visual tracker can be used. One is supplied by OpenCV-Contrib and is denoted as *KCF*. This one should work out of the box.
+The second implementation is denoted as *KCF2*. This one is needed to reproduce the reported results. In order to use this you need to install [KCFcpp-py-wrapper](https://github.com/uoip/KCFcpp-py-wrapper),
+which is a Cython based wrapper of the original KCF code for python.
+It is recommended to build and install OpenCV from scratch instead of using the PyPI package in order to have all the necessary headers and libraries available.
+
+Why? Because this implementation seems to work better and much faster than the one provided by OpenCV-Contrib.
+
+If you do not install this module the tracker will automatically fall back on the OpenCV implementation (i.e. *KCF*) and a warning is printed.
+You can get rid of this warning by either by installing the other KCF module as described above or explicitly request *KCF* instead of *KCF2* in the scripts.
+Note that the tracking performance will be affected by this.
-## Demo
+
+### Demo
Several demo scripts are included to reproduce the reported results on the [UA-DETRAC](http://detrac-db.rit.albany.edu/)
-and the [MOT](https://motchallenge.net/) 16/17 benchmarks.
+, [MOT](https://motchallenge.net/) 16/17/19 and [VisDrone](http://www.aiskyeye.com/) benchmarks.
-Basic demo script:
+Basic demo script (not dataset specific, can be used for other applications):
```
$ ./demo.py -h
-usage: demo.py [-h] -d DETECTION_PATH -o OUTPUT_PATH [-sl SIGMA_L]
- [-sh SIGMA_H] [-si SIGMA_IOU] [-tm T_MIN]
+usage: demo.py [-h] [-v VISUAL] [-hr KEEP_UPPER_HEIGHT_RATIO] [-f FRAMES_PATH]
+ -d DETECTION_PATH -o OUTPUT_PATH [-sl SIGMA_L] [-sh SIGMA_H]
+ [-si SIGMA_IOU] [-tm T_MIN] [-ttl TTL] [-nms NMS] [-fmt FORMAT]
-IOU Tracker demo script
+IOU/V-IOU Tracker demo script
optional arguments:
-h, --help show this help message and exit
+ -v VISUAL, --visual VISUAL
+ visual tracker for V-IOU. Currently supported are
+ [BOOSTING, MIL, KCF, KCF2, TLD, MEDIANFLOW, GOTURN,
+ NONE] see README.md for furthert details
+ -hr KEEP_UPPER_HEIGHT_RATIO, --keep_upper_height_ratio KEEP_UPPER_HEIGHT_RATIO
+ Ratio of height of the object to track to the total
+ height of the object for visual tracking. e.g. upper
+ 30%
+ -f FRAMES_PATH, --frames_path FRAMES_PATH
+ sequence frames with format
+ '/path/to/frames/frame_{:04d}.jpg' where '{:04d}' will
+ be replaced with the frame id. (zero_padded to 4
+ digits, use {:05d} for 5 etc.)
-d DETECTION_PATH, --detection_path DETECTION_PATH
full path to CSV file containing the detections
-o OUTPUT_PATH, --output_path OUTPUT_PATH
output path to store the tracking results (MOT
- challenge devkit compatible format)
+ challenge/Visdrone devkit compatible format)
-sl SIGMA_L, --sigma_l SIGMA_L
low detection threshold
-sh SIGMA_H, --sigma_h SIGMA_H
@@ -44,16 +115,25 @@ optional arguments:
intersection-over-union threshold
-tm T_MIN, --t_min T_MIN
minimum track length
+ -ttl TTL, --ttl TTL time to live parameter for v-iou
+ -nms NMS, --nms NMS nms for loading multi-class detections
+ -fmt FORMAT, --format FORMAT
+ format of the detections [motchallenge, visdrone]
```
-Example for the MOT17-04 sequence (detections can be downloaded [here](https://motchallenge.net/data/MOT17/)):
+Example for the IOU tracker on the MOT17-04 sequence (detections can be downloaded [here](https://motchallenge.net/data/MOT17/)):
```
./demo.py -d ../mot17/train/MOT17-04-SDP/det/det.txt -o res/iou-tracker/MOT17-04-SDP.txt
```
+Example for the V-IOU tracker on the uav0000137_00458_v Visdrone sequence:
+```
+demo.py -f '/path/to/VisDrone2018-MOT-val/sequences/uav0000137_00458_v/{:07d}.jpg' -d /path/to/VisDrone2018-MOT-val/detections/uav0000137_00458_v.txt -o results/VisDrone2018-MOT-val/uav0000137_00458_v.txt -v MEDIANFLOW -sl 0.9 -sh 0.98 -si 0.1 -tm 23 --ttl 8 --nms 0.6 -fmt visdrone
+```
+
### DETRAC
To reproduce the reported results, download and extract the [DETRAC-toolkit](http://detrac-db.rit.albany.edu/download)
-and the detections you want to evaluate. Download links for the EB detections are provided below.
+and the detections you want to evaluate. Download links for the EB and Mask R-CNN detections are provided below.
Clone this repository into "DETRAC-MOT-toolkit/trackers/".
Follow the instructions to configure the toolkit for tracking evaluation and set the tracker name in "DETRAC_experiment.m":
@@ -62,32 +142,48 @@ tracker.trackerName = 'iou-tracker';
```
and run the script.
+You can switch between IOU and V-IOU and select the different parameters for different detectors in run_tracker.m
-Note that you still need a working python environment with numpy installed.
+Note that you still need a working python environment with numpy for IOU and all other dependencies for V-IOU installed.
You should obtain something like the following results for the 'DETRAC-Train' set:
##### DETRAC-Train Results
-| Detector | PR-Rcll | PR-Prcn | PR-FAR | PR-MT | PR-PT | PR-ML | PR-FP | PR-FN | PR-IDs| PR-FM | PR-MOTA | PR-MOTP | PR-MOTAL |
-| -------- | ------- | ------- | ------ | ----- | ------ | ----- | ------- | ------- | ----- | ----- | ------- | ------- | -------- |
-| EB |37.86 |44.73 |0.10 |32.34 |12.88 |20.93 |7958.82 |163739.85|4129.40|4221.89|35.77 |40.81 |36.48 |
-| R-CNN |27.86 |52.90 |0.11 |19.53 |17.03 |18.56 |9047.95 |157521.18|4842.18|4969.57|25.46 |44.39 |26.29 |
-| CompACT |25.15 |49.64 |0.09 |18.40 |14.15 |18.91 |7681.50 |152078.88|2177.44|2282.27|23.44 |42.88 |23.8191 |
-| ACF |27.39 |52.68 |0.14 |20.24 |15.66 |19.40 |11553.49 |161293.27|1845.49|2101.44|25.07 |44.71 |25.39 |
+IOU Tracker:
+
+| Detector | PR-Rcll | PR-Prcn | PR-FAR | PR-MT | PR-PT | PR-ML | PR-FP | PR-FN | PR-IDs| PR-FM | PR-MOTA | PR-MOTP | PR-MOTAL |
+| -------- | ------- | ------- | ------ | ----- | ------ | ----- | ------- | ------- | ----- | ----- | ------- | ------- | -------- |
+| EB |37.86 |44.73 |0.10 |32.34 |12.88 |20.93 |7958.82 |163739.85|4129.40|4221.89|35.77 |40.81 |36.48 |
+| R-CNN |27.86 |52.90 |0.11 |19.53 |17.03 |18.56 |9047.95 |157521.18|4842.18|4969.57|25.46 |44.39 |26.29 |
+| CompACT |25.20 |49.69 |0.10 |18.50 |14.11 |19.06 |8053.54 |153026.99|2021.84|2302.83|23.46 |42.96 |23.81 |
+| ACF |27.39 |52.68 |0.14 |20.24 |15.66 |19.40 |11553.49 |161293.27|1845.49|2101.44|25.07 |44.71 |25.39 |
+| Mask R-CNN |43.21 |47.26 |0.60 |37.22 |11.46 |24.24 |50096.88 |171714.09|1021.94|929.53 |34.36 |45.43 |34.54 |
+
+V-IOU Tracker:
+
+| Detector | PR-Rcll | PR-Prcn | PR-FAR | PR-MT | PR-PT | PR-ML | PR-FP | PR-FN | PR-IDs| PR-FM | PR-MOTA | PR-MOTP | PR-MOTAL |
+| -------- | ------- | ------- | ------ | ----- | ------ | ----- | ------- | ------- | ----- | ----- | ------- | ------- | -------- |
+| CompACT |26.84 |49.57 |0.10 |19.63 |14.67 |17.39 |8750.71 |143532.90|244.98 |444.20 |25.29 |41.58 |25.33 |
+| Mask R-CNN |42.80 |47.50 |0.60 |38.32 |8.36 |26.24 |50294.76 |174052.00|448.16 |293.69 |34.02 |46.87 |34.10 |
+
##### DETRAC-Test (Overall) Results
The reference results are taken from the [UA-DETRAC results](http://detrac-db.rit.albany.edu/TraRet) site. Only the best tracker / detector
combination is displayed for each reference method.
-| Tracker | Detector | PR-MOTA | PR-MOTP | PR-MT | PR-ML | PR-IDs | PR-FM | PR-FP | PR-FN | Speed |
-| ------------- | -------- | ------- | ----------- | --------- | --------- | -------- | -------- | ---------- | ---------- | -------------- |
-|CEM | CompACT | 5.1\% |35.2\% |3.0\% |35.3\% |**267.9** |**352.3** |**12341.2** |260390.4 |4.62 fps |
-|CMOT | CompACT | 12.6\% |36.1\% |16.1\% |18.6\% |285.3 |1516.8 |57885.9 |**167110.8**| & 3.79 fps |
-|GOG | CompACT | 14.2\% |37.0\% |13.9\% |19.9\% |3334.6 |3172.4 |32092.9 |180183.8 |390 fps |
-|DCT | R-CNN | 11.7\% |38.0\% |10.1\% |22.8\% |758.7 |742.9 |336561.2 |210855.6 |0.71 fps |
-|H2T | CompACT | 12.4\% |35.7\% |14.8\% |19.4\% |852.2 |1117.2 |51765.7 |173899.8 | 3.02 fps |
-|IHTLS | CompACT | 11.1\% |36.8\% |13.8\% |19.9\% |953.6 |3556.9 |53922.3 |180422.3 |19.79 fps |
-|**IOU** | R-CNN |16.0\% |**38.3\%** |13.8\% |20.7\% |5029.4 |5795.7 |22535.1 |193041.9 |**100,840 fps** |
-|**IOU** | EB |**19.4\%** |28.9\% |**17.7\%** |**18.4\%** |2311.3 |2445.9 |14796.5 |171806.8 |6,902 fps |
+| Tracker | Detector | PR-MOTA | PR-MOTP | PR-MT | PR-ML | PR-IDs | PR-FM | PR-FP | PR-FN | Speed |
+| ------------- | ----------- | ------- | ----------- | --------- | --------- | -------- | -------- | ---------- | ---------- | -------------- |
+|CEM | CompACT | 5.1\% |35.2\% |3.0\% |35.3\% |267.9 |352.3 |**12341.2** |260390.4 |4.62 fps |
+|CMOT | CompACT | 12.6\% |36.1\% |16.1\% |18.6\% |285.3 |1516.8 |57885.9 |167110.8 |3.79 fps |
+|GOG | CompACT | 14.2\% |37.0\% |13.9\% |19.9\% |3334.6 |3172.4 |32092.9 |180183.8 |390 fps |
+|DCT | R-CNN | 11.7\% |38.0\% |10.1\% |22.8\% |758.7 |742.9 |336561.2 |210855.6 |0.71 fps |
+|H2T | CompACT | 12.4\% |35.7\% |14.8\% |19.4\% |852.2 |1117.2 |51765.7 |173899.8 |3.02 fps |
+|IHTLS | CompACT | 11.1\% |36.8\% |13.8\% |19.9\% |953.6 |3556.9 |53922.3 |180422.3 |19.79 fps |
+|**IOU** | R-CNN |16.0\% |**38.3\%** |13.8\% |20.7\% |5029.4 |5795.7 |22535.1 |193041.9 |100,840 fps |
+|**IOU** | EB |19.4\% |28.9\% |17.7\% |**18.4\%** |2311.3 |2445.9 |14796.5 |171806.8 |6,902 fps |
+|**IOU** | CompACT | 16.1\% |37.0\% |14.8\% |19.7\% |2308.1 |3250.4 |24349.4 |176752.8 |**327,660 fps** |
+|**IOU** | Mask R-CNN | **30.7\%**|37.0\% |30.3\% |21.5\% |668.0 |733.6 |17370.3 |179505.9 |14,956 fps |
+|**V-IOU** | CompACT | 17.7\% |36.4\% |17.4\% |18.8\% |363.8 |1123.5 |26413.3 |**166571.7**|1117.90fps |
+|**V-IOU** | Mask R-CNN | **30.7\%**|37.0\% |**32.0\%** |22.6\% |**162.6** |**286.2** |18046.2 |179191.2 |359.18 fps |
##### EB detections
The public detections of [EB](http://zyb.im/research/EB/) are not available on the
@@ -97,6 +193,72 @@ publication are available here:
* [EB Train](https://tubcloud.tu-berlin.de/s/EtC6cFEYsAU0gFQ/download)
* [EB Test](https://tubcloud.tu-berlin.de/s/oKM3dYhJbMFl1dY/download)
+##### Mask R-CNN detections
+These detections are generated using a recent Mask R-CNN implementation trained on COCO.
+Only bounding boxes for COCOs *car*, *bus* and *truck* classes are included.
+Note that the detector is called "frcnn" (use `options.detectorSet = {'frcnn'};` in *initialize_environment.m*).
+* [Mask R-CNN Train](https://tubcloud.tu-berlin.de/s/MnGRGdH98WY9xQr/download)
+* [Mask R-CNN Test](https://tubcloud.tu-berlin.de/s/EztsFgm5AL8Jwtt/download)
+
+## Motchallenge
+
+### MOT16
+To reproduce the reported [MOT16 results](https://motchallenge.net/results/MOT16/) of the paper, use the mot16.py script:
+
+```
+$ ./mot16.py -h
+usage: mot16.py [-h] -m SEQMAP -o RES_DIR -b BENCHMARK_DIR [-sl SIGMA_L]
+ [-sh SIGMA_H] [-si SIGMA_IOU] [-tm T_MIN]
+
+IOU Tracker MOT demo script. Default parameters are set to reproduce the
+results using the SDP detections.
+
+optional arguments:
+ -h, --help show this help message and exit
+ -m SEQMAP, --seqmap SEQMAP
+ full path to the seqmap file to evaluate
+ -o RES_DIR, --res_dir RES_DIR
+ path to the results directory
+ -b BENCHMARK_DIR, --benchmark_dir BENCHMARK_DIR
+ path to the sequence directory
+ -sl SIGMA_L, --sigma_l SIGMA_L
+ low detection threshold
+ -sh SIGMA_H, --sigma_h SIGMA_H
+ high detection threshold
+ -si SIGMA_IOU, --sigma_iou SIGMA_IOU
+ intersection-over-union threshold
+ -tm T_MIN, --t_min T_MIN
+ minimum track length
+```
+
+Examples (you will probably need to adapt the paths):
+```
+# SDP:
+./mot16.py -m ../motchallenge/seqmaps/sdp-train.txt -o ../motchallenge/res/MOT16/iou-tracker -b ../data/mot17/train
+
+# FRCNN:
+./mot16.py -m ../motchallenge/seqmaps/frcnn-train.txt -o ../motchallenge/res/MOT16/iou-tracker -b ../data/mot17/train -sl 0 -sh 0.9 -si 0.3 -tm 5
+```
+
+The seqmap files can be found under "seqmaps" and need to be copied to the respective directory of the
+motchallenge devkit.
+You should obtain something like the following results for the train set:
+
+##### MOT16 Train Results
+| Detector | IDF1 | IDP | IDR | Rcll | Prcn | FAR | GT | MT | PT | ML | FP | FN | IDs | FM | MOTA | MOTP | MOTAL |
+| -------- | ---- | --- | --- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | ---- | ----- |
+|SDP |24.7 |46.2 |16.9 |65.0 |97.6 |0.34 |546 |178 |232 |136 |1796 |39348|1198 |1453 |62.3 |83.4 |63.4 |
+|FRCNN |21.0 |46.5 |13.6 |51.8 |97.2 |0.31 |546 |109 |261 |176 |1674 |54082|716 |810 |49.7 |88.2 |50.3 |
+
+##### MOT16 Test Results
+| Detector | Rcll | Prcn | FAR | GT | MT | PT | ML | FP | FN | IDs | FM | MOTA | MOTP |
+| -------- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | ---- |
+|SDP |61.5 |95.2 |0.96 |759 |179 |330 |250 |5702 |70278|2167 |3028 |57.1 |77.1 |
+|FRCNN |50.9 |92.4 |1.29 |759 |113 |381 |265 |7639 |89535| 2284|2310 |45.4 |77.5 |
+
+ Please note that this evaluation already includes the new ground truth of the MOT17 release.
+
+
### MOT17
The IOU Tracker was evaluated on the MOT17 benchmark as well. To determine the best parameters for each detector, an
exhaustive search of the parameter space was performed similar to the one of the MOT16 evaluation reported in the paper.
@@ -145,61 +307,55 @@ Examples (you will probably need to adapt the paths):
| ---- | ---- | --- | ----- | ----- | ----- | ------ | ------ | ---- |
| 45.5 |76.9 |1.1 |15.7\% |40.5\% |19,993 |281,643 | 5,988 |7,404 |
-### MOT16
-To reproduce the reported [MOT16 results](https://motchallenge.net/results/MOT16/) of the paper, use the mot16.py script:
-
-```
-$ ./mot16.py -h
-usage: mot16.py [-h] -m SEQMAP -o RES_DIR -b BENCHMARK_DIR [-sl SIGMA_L]
- [-sh SIGMA_H] [-si SIGMA_IOU] [-tm T_MIN]
-IOU Tracker MOT demo script. Default parameters are set to reproduce the
-results using the SDP detections.
+### CVPR19
+To reproduce the results on the CVPR19 dataset you can use the cvpr19.sh bash script.
-optional arguments:
- -h, --help show this help message and exit
- -m SEQMAP, --seqmap SEQMAP
- full path to the seqmap file to evaluate
- -o RES_DIR, --res_dir RES_DIR
- path to the results directory
- -b BENCHMARK_DIR, --benchmark_dir BENCHMARK_DIR
- path to the sequence directory
- -sl SIGMA_L, --sigma_l SIGMA_L
- low detection threshold
- -sh SIGMA_H, --sigma_h SIGMA_H
- high detection threshold
- -si SIGMA_IOU, --sigma_iou SIGMA_IOU
- intersection-over-union threshold
- -tm T_MIN, --t_min T_MIN
- minimum track length
+Edit the first lines according to your setup:
```
-
-Examples (you will probably need to adapt the paths):
+# set these variables according to your setup
+seq_dir=/path/to/cvpr19/train # base directory of the split (cvpr19/train, cvpr19/test etc.)
+results_dir=results/cvpr19 # output directory, will be created if not existing
```
-# SDP:
-./mot16.py -m ../motchallenge/seqmaps/sdp-train.txt -o ../motchallenge/res/MOT16/iou-tracker -b ../data/mot17/train
-
-# FRCNN:
-./mot16.py -m ../motchallenge/seqmaps/frcnn-train.txt -o ../motchallenge/res/MOT16/iou-tracker -b ../data/mot17/train -sl 0 -sh 0.9 -si 0.3 -tm 5
+Then, run (The used parameters for the *demo.py* script will be displayed for your convenience):
+```
+./cvpr19.sh
```
-The seqmap files can be found under "seqmaps" and need to be copied to the respective directory of the
-motchallenge devkit.
-You should obtain something like the following results for the train set:
+Note that this requires to have *KCF2* tracker installed.
-##### MOT16 Train Results
+Note that only the upper 30% of the detections are used for visual tracking since the bottom part is often occluded.
+Due to time constraints only *KCF2* was evaluated for the CVPR19 challenge participation,
+*MEDIANFLOW* might yield better results like for the [Visdrone-MOT](#Visdrone-MOT) experiments but additional parameter tuning is required.
+
+##### CVPR19 Train Results
| Detector | IDF1 | IDP | IDR | Rcll | Prcn | FAR | GT | MT | PT | ML | FP | FN | IDs | FM | MOTA | MOTP | MOTAL |
| -------- | ---- | --- | --- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | ---- | ----- |
-|SDP |24.7 |46.2 |16.9 |65.0 |97.6 |0.34 |546 |178 |232 |136 |1796 |39348|1198 |1453 |62.3 |83.4 |63.4 |
-|FRCNN |21.0 |46.5 |13.6 |51.8 |97.2 |0.31 |546 |109 |261 |176 |1674 |54082|716 |810 |49.7 |88.2 |50.3 |
+| FRCNN | 53.9 | 71.6| 43.2| 59.9 | 99.3 | 0.63| 2274| 670 |1187 | 417 |55844|97170| 3272|4792 | 59.2 | 87.5 | 59.5 |
-##### MOT16 Test Results
-| Detector | Rcll | Prcn | FAR | GT | MT | PT | ML | FP | FN | IDs | FM | MOTA | MOTP |
-| -------- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | ---- |
-|SDP |61.5 |95.2 |0.96 |759 |179 |330 |250 |5702 |70278|2167 |3028 |57.1 |77.1 |
-|FRCNN |50.9 |92.4 |1.29 |759 |113 |381 |265 |7639 |89535| 2284|2310 |45.4 |77.5 |
+### Visdrone-MOT
+To reproduce the results on the Visdrone MOT dataset you can use the visdrone-mot.sh bash script.
+Edit the first lines according to your setup:
+```
+# set these variables according to your setup
+visdrone_dir=/path/to/VisDrone2018-MOT-val # base directory of the split (VisDrone2018-MOT-val, VisDrone2018-MOT-train etc.)
+results_dir=results/VisDrone2018-MOT-val # output directory, will be created if not existing
+vis_tracker=MEDIANFLOW # [MEDIANFLOW, KCF2, NONE] parameter set as used in the paper
+```
+Then, run (The used parameters for the *demo.py* script will be displayed for your convenience):
+```
+./visdrone-mot.sh
+```
+
+##### Visdrone-MOT Val Results
+For the *VisDrone2018-MOT-val* split you should get the following results:
+
+| Visual Tracker | IDF1 | IDP | IDR | Rcll | Prcn | FAR | GT | MT | PT | ML | FP | FN | IDs | FM | MOTA | MOTP | MOTAL |
+| -------------- | ---- | --- | --- | ---- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---- | ---- | ----- |
+| None | 40.9 | 68.5| 29.2| 34.6 | 81.3 | 0.60| 476 | 102 | 76 | 297 | 5736|46979| 177 | 435 | 26.4 | 78.1 | 26.6 |
+| KCF | 45.3 | 75.3| 32.4| 35.2 | 81.8 | 0.59| 476 | 105 | 64 | 305 | 5605|46578| 76 | 385 | 27.3 | 77.9 | 27.4 |
+| Medianflow | 45.6 | 75.9| 32.6| 35.3 | 82.2 | 0.58| 476 | 107 | 63 | 304 | 5494|46466| 65 | 378 | 27.6 | 77.8 | 27.7 |
- Please note that this evaluation already includes the new ground truth of the MOT17 release.
## Contact
If you have any questions or encounter problems regarding the method/code feel free to contact me
diff --git a/cvpr19.sh b/cvpr19.sh
new file mode 100755
index 0000000..ed06e2f
--- /dev/null
+++ b/cvpr19.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# set these variables according to your setup
+seq_dir=/path/to/cvpr19/train # base directory of the split (cvpr19/train, cvpr19/test etc.)
+results_dir=results/cvpr19 # output directory, will be created if not existing
+
+
+mkdir -p ${results_dir}
+
+options="-v KCF2 -sl 0.3 -sh 0.8 -si 0.4 -tm 5 --ttl 20 -hr 0.3 -fmt motchallenge"
+for seq in $(ls $seq_dir); do
+ echo $seq
+ python demo.py -f ${seq_dir}/${seq}/img1/{:06d}.jpg -d ${seq_dir}/${seq}/det/det.txt \
+ -o ${results_dir}/${seq}.txt ${options}
+done
diff --git a/demo.py b/demo.py
index 21dde6b..3041970 100755
--- a/demo.py
+++ b/demo.py
@@ -7,33 +7,53 @@
# Written by Erik Bochinski
# ---------------------------------------------------------
-from time import time
import argparse
from iou_tracker import track_iou
+from viou_tracker import track_viou
from util import load_mot, save_to_csv
def main(args):
- detections = load_mot(args.detection_path)
+ formats = ['motchallenge', 'visdrone']
+ assert args.format in formats, "format '{}' unknown supported formats are: {}".format(args.format, formats)
- start = time()
- tracks = track_iou(detections, args.sigma_l, args.sigma_h, args.sigma_iou, args.t_min)
- end = time()
+ with_classes = False
+ if args.format == 'visdrone':
+ with_classes = True
+ detections = load_mot(args.detection_path, nms_overlap_thresh=args.nms, with_classes=with_classes)
- num_frames = len(detections)
- print("finished at " + str(int(num_frames / (end - start))) + " fps!")
+ if args.visual:
+ tracks = track_viou(args.frames_path, detections, args.sigma_l, args.sigma_h, args.sigma_iou, args.t_min,
+ args.ttl, args.visual, args.keep_upper_height_ratio)
+ else:
+ if with_classes:
+ # track_viou can also be used without visual tracking, but note that the speed will be much slower compared
+ # to track_iou. However, this way supports the optimal LAP solving and the handling of multiple object classes:
+ tracks = track_viou(args.frames_path, detections, args.sigma_l, args.sigma_h, args.sigma_iou, args.t_min,
+ args.ttl, 'NONE', args.keep_upper_height_ratio)
+ else:
+ tracks = track_iou(detections, args.sigma_l, args.sigma_h, args.sigma_iou, args.t_min)
- save_to_csv(args.output_path, tracks)
+ save_to_csv(args.output_path, tracks, fmt=args.format)
if __name__ == '__main__':
-
- parser = argparse.ArgumentParser(description="IOU Tracker demo script")
+ parser = argparse.ArgumentParser(description="IOU/V-IOU Tracker demo script")
+ parser.add_argument('-v', '--visual', type=str, help="visual tracker for V-IOU. Currently supported are "
+ "[BOOSTING, MIL, KCF, KCF2, TLD, MEDIANFLOW, GOTURN, NONE] "
+ "see README.md for furthert details")
+ parser.add_argument('-hr', '--keep_upper_height_ratio', type=float, default=1.0,
+ help="Ratio of height of the object to track to the total height of the object "
+ "for visual tracking. e.g. upper 30%%")
+ parser.add_argument('-f', '--frames_path', type=str,
+ help="sequence frames with format '/path/to/frames/frame_{:04d}.jpg' where '{:04d}' will "
+ "be replaced with the frame id. (zero_padded to 4 digits, use {:05d} for 5 etc.)")
parser.add_argument('-d', '--detection_path', type=str, required=True,
help="full path to CSV file containing the detections")
parser.add_argument('-o', '--output_path', type=str, required=True,
- help="output path to store the tracking results (MOT challenge devkit compatible format)")
+ help="output path to store the tracking results "
+ "(MOT challenge/Visdrone devkit compatible format)")
parser.add_argument('-sl', '--sigma_l', type=float, default=0,
help="low detection threshold")
parser.add_argument('-sh', '--sigma_h', type=float, default=0.5,
@@ -42,6 +62,17 @@ def main(args):
help="intersection-over-union threshold")
parser.add_argument('-tm', '--t_min', type=float, default=2,
help="minimum track length")
+ parser.add_argument('-ttl', '--ttl', type=int, default=1,
+ help="time to live parameter for v-iou")
+ parser.add_argument('-nms', '--nms', type=float, default=None,
+ help="nms for loading multi-class detections")
+ parser.add_argument('-fmt', '--format', type=str, default='motchallenge',
+ help='format of the detections [motchallenge, visdrone]')
args = parser.parse_args()
+ assert not args.visual or args.visual and args.frames_path, "visual tracking requires video frames, " \
+ "please specify via --frames_path"
+
+ assert 0.0 < args.keep_upper_height_ratio <= 1.0, "only values between 0 and 1 are allowed"
+ assert args.nms is None or 0.0 <= args.nms <= 1.0, "only values between 0 and 1 are allowed"
main(args)
diff --git a/mot16.py b/mot16.py
index de8f42a..b7bf173 100755
--- a/mot16.py
+++ b/mot16.py
@@ -25,7 +25,7 @@ def main(args):
det_path = args.benchmark_dir + "/" + seq + "/det/det.txt"
out_path = args.res_dir + "/" + seq + ".txt"
- detections = load_mot(det_path)
+ detections = load_mot(det_path, with_classes=False)
start = time()
tracks = track_iou(detections, args.sigma_l, args.sigma_h, args.sigma_iou, args.t_min)
diff --git a/mot17.py b/mot17.py
index be179f0..1da6b8d 100755
--- a/mot17.py
+++ b/mot17.py
@@ -45,7 +45,7 @@ def main(args):
det_path = args.benchmark_dir + "/" + seq + "/det/det.txt"
out_path = args.res_dir + "/" + seq + ".txt"
- detections = load_mot(det_path)
+ detections = load_mot(det_path, with_classes=False)
start = time()
tracks = track_iou(detections, sigma_l, sigma_h, sigma_iou, t_min)
diff --git a/run_tracker.m b/run_tracker.m
index 6188220..86fb9d1 100644
--- a/run_tracker.m
+++ b/run_tracker.m
@@ -1,10 +1,37 @@
function [stateInfo, speed] = run_tracker(curSequence, baselinedetections)
%% tracker configuration
-%% R-CNN
+ttl = 0;
+tracker_type = '';
+
+%% v-iou tracker configurations
+% %% Mask R-CNN (frcnn)
sigma_l = 0;
-sigma_h = 0.7;
-sigma_iou = 0.5;
-t_min = 2;
+sigma_h = 0.98;
+sigma_iou = 0.6;
+t_min = 13;
+ttl=6;
+tracker_type='KCF2';
+
+% %% CompACT
+%sigma_l = 0;
+%sigma_h = 0.3;
+%sigma_iou = 0.5;
+%t_min = 3;
+%ttl=12;
+%tracker_type='KCF2';
+
+%% iou tracker configurations
+% %% Mask R-CNN (frcnn)
+%sigma_l = 0;
+%sigma_h = 0.95;
+%sigma_iou = 0.6;
+%t_min = 7;
+
+% %% R-CNN
+% sigma_l = 0;
+% sigma_h = 0.7;
+% sigma_iou = 0.5;
+% t_min = 2;
% %% ACF
% sigma_l = 0;
@@ -15,7 +42,7 @@
% %% CompACT
% sigma_l = 0;
% sigma_h = 0.2;
-% sigma_iou = 0.5;
+% sigma_iou = 0.4;
% t_min = 2;
% %% EB
@@ -26,8 +53,11 @@
%% running tracking algorithm
try
- ret = py.iou_tracker.track_iou_matlab_wrapper(py.numpy.array(baselinedetections(:).'), sigma_l, sigma_h, sigma_iou, t_min);
-
+ if strcmp(tracker_type, '')
+ ret = py.iou_tracker.track_iou_matlab_wrapper(py.numpy.array(baselinedetections(:).'), sigma_l, sigma_h, sigma_iou, t_min);
+ else
+ ret = py.viou_tracker.track_viou_matlab_wrapper(curSequence.imgFolder, py.numpy.array(baselinedetections(:).'), sigma_l, sigma_h, sigma_iou, t_min, ttl, tracker_type);
+ end
catch exception
disp('error while calling the python tracking module: ')
disp(' ')
diff --git a/util.py b/util.py
index f7d5cd0..c6717b4 100644
--- a/util.py
+++ b/util.py
@@ -7,23 +7,36 @@
import numpy as np
import csv
+import os
-def load_mot(detections):
+visdrone_classes = {'car': 4, 'bus': 9, 'truck': 6, 'pedestrian': 1, 'van': 5}
+
+
+def load_mot(detections, nms_overlap_thresh=None, with_classes=True, nms_per_class=False):
"""
Loads detections stored in a mot-challenge like formatted CSV or numpy array (fieldNames = ['frame', 'id', 'x', 'y',
'w', 'h', 'score']).
Args:
- detections
+ detections (str, numpy.ndarray): path to csv file containing the detections or numpy array containing them.
+ nms_overlap_thresh (float, optional): perform non-maximum suppression on the input detections with this thrshold.
+ no nms is performed if this parameter is not specified.
+ with_classes (bool, optional): indicates if the detections have classes or not. set to false for motchallange.
+ nms_per_class (bool, optional): perform non-maximum suppression for each class separately
Returns:
list: list containing the detections for each frame.
"""
+ if nms_overlap_thresh:
+ assert with_classes, "currently only works with classes available"
data = []
if type(detections) is str:
raw = np.genfromtxt(detections, delimiter=',', dtype=np.float32)
+ if np.isnan(raw).all():
+ raw = np.genfromtxt(detections, delimiter=' ', dtype=np.float32)
+
else:
# assume it is an array
assert isinstance(detections, np.ndarray), "only numpy arrays or *.csv paths are supported as detections."
@@ -34,16 +47,127 @@ def load_mot(detections):
idx = raw[:, 0] == i
bbox = raw[idx, 2:6]
bbox[:, 2:4] += bbox[:, 0:2] # x1, y1, w, h -> x1, y1, x2, y2
+ bbox -= 1 # correct 1,1 matlab offset
scores = raw[idx, 6]
+
+ if with_classes:
+ classes = raw[idx, 7]
+
+ bbox_filtered = None
+ scores_filtered = None
+ classes_filtered = None
+ for coi in visdrone_classes:
+ cids = classes==visdrone_classes[coi]
+ if nms_per_class and nms_overlap_thresh:
+ bbox_tmp, scores_tmp = nms(bbox[cids], scores[cids], nms_overlap_thresh)
+ else:
+ bbox_tmp, scores_tmp = bbox[cids], scores[cids]
+
+ if bbox_filtered is None:
+ bbox_filtered = bbox_tmp
+ scores_filtered = scores_tmp
+ classes_filtered = [coi]*bbox_filtered.shape[0]
+ elif len(bbox_tmp) > 0:
+ bbox_filtered = np.vstack((bbox_filtered, bbox_tmp))
+ scores_filtered = np.hstack((scores_filtered, scores_tmp))
+ classes_filtered += [coi] * bbox_tmp.shape[0]
+
+ if bbox_filtered is not None:
+ bbox = bbox_filtered
+ scores = scores_filtered
+ classes = classes_filtered
+
+ if nms_per_class is False and nms_overlap_thresh:
+ bbox, scores, classes = nms(bbox, scores, nms_overlap_thresh, np.array(classes))
+
+ else:
+ classes = ['pedestrian']*bbox.shape[0]
+
dets = []
- for bb, s in zip(bbox, scores):
- dets.append({'bbox': (bb[0], bb[1], bb[2], bb[3]), 'score': s})
+ for bb, s, c in zip(bbox, scores, classes):
+ dets.append({'bbox': (bb[0], bb[1], bb[2], bb[3]), 'score': s, 'class': c})
data.append(dets)
return data
-def save_to_csv(out_path, tracks):
+def nms(boxes, scores, overlapThresh, classes=None):
+ """
+ perform non-maximum suppression. based on Malisiewicz et al.
+ Args:
+ boxes (numpy.ndarray): boxes to process
+ scores (numpy.ndarray): corresponding scores for each box
+ overlapThresh (float): overlap threshold for boxes to merge
+ classes (numpy.ndarray, optional): class ids for each box.
+
+ Returns:
+ (tuple): tuple containing:
+
+ boxes (list): nms boxes
+ scores (list): nms scores
+ classes (list, optional): nms classes if specified
+ """
+ # # if there are no boxes, return an empty list
+ # if len(boxes) == 0:
+ # return [], [], [] if classes else [], []
+
+ # if the bounding boxes integers, convert them to floats --
+ # this is important since we'll be doing a bunch of divisions
+ if boxes.dtype.kind == "i":
+ boxes = boxes.astype("float")
+
+ if scores.dtype.kind == "i":
+ scores = scores.astype("float")
+
+ # initialize the list of picked indexes
+ pick = []
+
+ # grab the coordinates of the bounding boxes
+ x1 = boxes[:, 0]
+ y1 = boxes[:, 1]
+ x2 = boxes[:, 2]
+ y2 = boxes[:, 3]
+ #score = boxes[:, 4]
+ # compute the area of the bounding boxes and sort the bounding
+ # boxes by the bottom-right y-coordinate of the bounding box
+ area = (x2 - x1 + 1) * (y2 - y1 + 1)
+ idxs = np.argsort(scores)
+
+ # keep looping while some indexes still remain in the indexes
+ # list
+ while len(idxs) > 0:
+ # grab the last index in the indexes list and add the
+ # index value to the list of picked indexes
+ last = len(idxs) - 1
+ i = idxs[last]
+ pick.append(i)
+
+ # find the largest (x, y) coordinates for the start of
+ # the bounding box and the smallest (x, y) coordinates
+ # for the end of the bounding box
+ xx1 = np.maximum(x1[i], x1[idxs[:last]])
+ yy1 = np.maximum(y1[i], y1[idxs[:last]])
+ xx2 = np.minimum(x2[i], x2[idxs[:last]])
+ yy2 = np.minimum(y2[i], y2[idxs[:last]])
+
+ # compute the width and height of the bounding box
+ w = np.maximum(0, xx2 - xx1 + 1)
+ h = np.maximum(0, yy2 - yy1 + 1)
+
+ # compute the ratio of overlap
+ overlap = (w * h) / area[idxs[:last]]
+
+ # delete all indexes from the index list that have
+ idxs = np.delete(idxs, np.concatenate(([last],
+ np.where(overlap > overlapThresh)[0])))
+
+ if classes is not None:
+ return boxes[pick], scores[pick], classes[pick]
+ else:
+ return boxes[pick], scores[pick]
+
+
+def save_to_csv(out_path, tracks, fmt='motchallenge'):
"""
Saves tracks to a CSV file.
@@ -51,9 +175,14 @@ def save_to_csv(out_path, tracks):
out_path (str): path to output csv file.
tracks (list): list of tracks to store.
"""
-
+ os.makedirs(os.path.dirname(out_path), exist_ok=True)
with open(out_path, "w") as ofile:
- field_names = ['frame', 'id', 'x', 'y', 'w', 'h', 'score', 'wx', 'wy', 'wz']
+ if fmt == 'motchallenge':
+ field_names = ['frame', 'id', 'x', 'y', 'w', 'h', 'score', 'wx', 'wy', 'wz']
+ elif fmt == 'visdrone':
+ field_names = ['frame', 'id', 'x', 'y', 'w', 'h', 'score', 'object_category', 'truncation', 'occlusion']
+ else:
+ raise ValueError("unknown format type '{}'".format(fmt))
odict = csv.DictWriter(ofile, field_names)
id_ = 1
@@ -61,14 +190,21 @@ def save_to_csv(out_path, tracks):
for i, bbox in enumerate(track['bboxes']):
row = {'id': id_,
'frame': track['start_frame'] + i,
- 'x': bbox[0],
- 'y': bbox[1],
+ 'x': bbox[0]+1,
+ 'y': bbox[1]+1,
'w': bbox[2] - bbox[0],
'h': bbox[3] - bbox[1],
- 'score': track['max_score'],
- 'wx': -1,
- 'wy': -1,
- 'wz': -1}
+ 'score': track['max_score']}
+ if fmt == 'motchallenge':
+ row['wx'] = -1
+ row['wy'] = -1
+ row['wz'] = -1
+ elif fmt == 'visdrone':
+ row['object_category'] = visdrone_classes[track['class']]
+ row['truncation'] = -1
+ row['occlusion'] = -1
+ else:
+ raise ValueError("unknown format type '{}'".format(fmt))
odict.writerow(row)
id_ += 1
diff --git a/viou_tracker.py b/viou_tracker.py
new file mode 100644
index 0000000..020a30e
--- /dev/null
+++ b/viou_tracker.py
@@ -0,0 +1,237 @@
+# ---------------------------------------------------------
+# IOU Tracker
+# Copyright (c) 2019 TU Berlin, Communication Systems Group
+# Licensed under The MIT License [see LICENSE for details]
+# Written by Erik Bochinski
+# ---------------------------------------------------------
+
+import cv2
+import numpy as np
+from lapsolver import solve_dense
+from tqdm import tqdm
+from time import time
+
+from util import iou, load_mot
+from vis_tracker import VisTracker
+
+
+def track_viou(frames_path, detections, sigma_l, sigma_h, sigma_iou, t_min, ttl, tracker_type, keep_upper_height_ratio):
+ """ V-IOU Tracker.
+ See "Extending IOU Based Multi-Object Tracking by Visual Information by E. Bochinski, T. Senst, T. Sikora" for
+ more information.
+
+ Args:
+ frames_path (str): path to ALL frames.
+ string must contain a placeholder like {:07d} to be replaced with the frame numbers.
+ detections (list): list of detections per frame, usually generated by util.load_mot
+ sigma_l (float): low detection threshold.
+ sigma_h (float): high detection threshold.
+ sigma_iou (float): IOU threshold.
+ t_min (float): minimum track length in frames.
+ ttl (float): maximum number of frames to perform visual tracking.
+ this can fill 'gaps' of up to 2*ttl frames (ttl times forward and backward).
+ tracker_type (str): name of the visual tracker to use. see VisTracker for more details.
+ keep_upper_height_ratio (float): float between 0.0 and 1.0 that determines the ratio of height of the object
+ to track to the total height of the object used for visual tracking.
+
+ Returns:
+ list: list of tracks.
+ """
+ if tracker_type == 'NONE':
+ assert ttl == 1, "ttl should not be larger than 1 if no visual tracker is selected"
+
+ tracks_active = []
+ tracks_extendable = []
+ tracks_finished = []
+ frame_buffer = []
+
+ for frame_num, detections_frame in enumerate(tqdm(detections), start=1):
+ # load frame and put into buffer
+ frame_path = frames_path.format(frame_num)
+ frame = cv2.imread(frame_path)
+ assert frame is not None, "could not read '{}'".format(frame_path)
+ frame_buffer.append(frame)
+ if len(frame_buffer) > ttl + 1:
+ frame_buffer.pop(0)
+
+ # apply low threshold to detections
+ dets = [det for det in detections_frame if det['score'] >= sigma_l]
+
+ track_ids, det_ids = associate(tracks_active, dets, sigma_iou)
+ updated_tracks = []
+ for track_id, det_id in zip(track_ids, det_ids):
+ tracks_active[track_id]['bboxes'].append(dets[det_id]['bbox'])
+ tracks_active[track_id]['max_score'] = max(tracks_active[track_id]['max_score'], dets[det_id]['score'])
+ tracks_active[track_id]['classes'].append(dets[det_id]['class'])
+ tracks_active[track_id]['det_counter'] += 1
+
+ if tracks_active[track_id]['ttl'] != ttl:
+ # reset visual tracker if active
+ tracks_active[track_id]['ttl'] = ttl
+ tracks_active[track_id]['visual_tracker'] = None
+
+ updated_tracks.append(tracks_active[track_id])
+
+ tracks_not_updated = [tracks_active[idx] for idx in set(range(len(tracks_active))).difference(set(track_ids))]
+
+ for track in tracks_not_updated:
+ if track['ttl'] > 0:
+ if track['ttl'] == ttl:
+ # init visual tracker
+ track['visual_tracker'] = VisTracker(tracker_type, track['bboxes'][-1], frame_buffer[-2],
+ keep_upper_height_ratio)
+ # viou forward update
+ ok, bbox = track['visual_tracker'].update(frame)
+
+ if not ok:
+ # visual update failed, track can still be extended
+ tracks_extendable.append(track)
+ continue
+
+ track['ttl'] -= 1
+ track['bboxes'].append(bbox)
+ updated_tracks.append(track)
+ else:
+ tracks_extendable.append(track)
+
+ # update the list of extendable tracks. tracks that are too old are moved to the finished_tracks. this should
+ # not be necessary but may improve the performance for large numbers of tracks (eg. for mot19)
+ tracks_extendable_updated = []
+ for track in tracks_extendable:
+ if track['start_frame'] + len(track['bboxes']) + ttl - track['ttl'] >= frame_num:
+ tracks_extendable_updated.append(track)
+ elif track['max_score'] >= sigma_h and track['det_counter'] >= t_min:
+ tracks_finished.append(track)
+ tracks_extendable = tracks_extendable_updated
+
+ new_dets = [dets[idx] for idx in set(range(len(dets))).difference(set(det_ids))]
+ dets_for_new = []
+
+ for det in new_dets:
+ finished = False
+ # go backwards and track visually
+ boxes = []
+ vis_tracker = VisTracker(tracker_type, det['bbox'], frame, keep_upper_height_ratio)
+
+ for f in reversed(frame_buffer[:-1]):
+ ok, bbox = vis_tracker.update(f)
+ if not ok:
+ # can not go further back as the visual tracker failed
+ break
+ boxes.append(bbox)
+
+ # sorting is not really necessary but helps to avoid different behaviour for different orderings
+ # preferring longer tracks for extension seems intuitive, LAP solving might be better
+ for track in sorted(tracks_extendable, key=lambda x: len(x['bboxes']), reverse=True):
+
+ offset = track['start_frame'] + len(track['bboxes']) + len(boxes) - frame_num
+ # association not optimal (LAP solving might be better)
+ # association is performed at the same frame, not adjacent ones
+ if 1 <= offset <= ttl - track['ttl'] and iou(track['bboxes'][-offset], bbox) >= sigma_iou:
+ if offset > 1:
+ # remove existing visually tracked boxes behind the matching frame
+ track['bboxes'] = track['bboxes'][:-offset+1]
+ track['bboxes'] += list(reversed(boxes))[1:]
+ track['bboxes'].append(det['bbox'])
+ track['max_score'] = max(track['max_score'], det['score'])
+ track['classes'].append(det['class'])
+ track['ttl'] = ttl
+ track['visual_tracker'] = None
+
+ tracks_extendable.remove(track)
+ if track in tracks_finished:
+ del tracks_finished[tracks_finished.index(track)]
+ updated_tracks.append(track)
+
+ finished = True
+ break
+ if finished:
+ break
+ if not finished:
+ dets_for_new.append(det)
+
+ # create new tracks
+ new_tracks = [{'bboxes': [det['bbox']], 'max_score': det['score'], 'start_frame': frame_num, 'ttl': ttl,
+ 'classes': [det['class']], 'det_counter': 1, 'visual_tracker': None} for det in dets_for_new]
+ tracks_active = []
+ for track in updated_tracks + new_tracks:
+ if track['ttl'] == 0:
+ tracks_extendable.append(track)
+ else:
+ tracks_active.append(track)
+
+ # finish all remaining active and extendable tracks
+ tracks_finished = tracks_finished + \
+ [track for track in tracks_active + tracks_extendable
+ if track['max_score'] >= sigma_h and track['det_counter'] >= t_min]
+
+ # remove last visually tracked frames and compute the track classes
+ for track in tracks_finished:
+ if ttl != track['ttl']:
+ track['bboxes'] = track['bboxes'][:-(ttl - track['ttl'])]
+ track['class'] = max(set(track['classes']), key=track['classes'].count)
+
+ del track['visual_tracker']
+
+ return tracks_finished
+
+
+def associate(tracks, detections, sigma_iou):
+ """ perform association between tracks and detections in a frame.
+ Args:
+ tracks (list): input tracks
+ detections (list): input detections
+ sigma_iou (float): minimum intersection-over-union of a valid association
+
+ Returns:
+ (tuple): tuple containing:
+
+ track_ids (numpy.array): 1D array with indexes of the tracks
+ det_ids (numpy.array): 1D array of the associated indexes of the detections
+ """
+ costs = np.empty(shape=(len(tracks), len(detections)), dtype=np.float32)
+ for row, track in enumerate(tracks):
+ for col, detection in enumerate(detections):
+ costs[row, col] = 1 - iou(track['bboxes'][-1], detection['bbox'])
+
+ np.nan_to_num(costs)
+ costs[costs > 1 - sigma_iou] = np.nan
+ track_ids, det_ids = solve_dense(costs)
+ return track_ids, det_ids
+
+
+def track_viou_matlab_wrapper(frames_path, detections, sigma_l, sigma_h, sigma_iou, t_min, ttl, tracker_type,
+ keep_upper_height_ratio=1.):
+ """
+ Matlab wrapper of the v-iou tracker for the detrac evaluation toolkit.
+
+ Args:
+ detections (numpy.array): numpy array of detections, usually supplied by run_tracker.m
+ sigma_l (float): low detection threshold.
+ sigma_h (float): high detection threshold.
+ sigma_iou (float): IOU threshold.
+ t_min (float): minimum track length in frames.
+
+ Returns:
+ float: speed in frames per second.
+ list: list of tracks.
+ """
+
+ detections = detections.reshape((7, -1)).transpose()
+ dets = load_mot(detections, with_classes=False)
+ start = time()
+ tracks = track_viou(frames_path+"img{:05d}.jpg", dets, sigma_l, sigma_h, sigma_iou, int(t_min), int(ttl), tracker_type, keep_upper_height_ratio)
+ end = time()
+
+ id_ = 1
+ out = []
+ for track in tracks:
+ for i, bbox in enumerate(track['bboxes']):
+ out += [float(bbox[0]), float(bbox[1]), float(bbox[2] - bbox[0]), float(bbox[3] - bbox[1]),
+ float(track['start_frame'] + i), float(id_)]
+ id_ += 1
+
+ num_frames = len(dets)
+ speed = num_frames / (end - start)
+
+ return speed, out
diff --git a/vis_tracker.py b/vis_tracker.py
new file mode 100644
index 0000000..0a11c8b
--- /dev/null
+++ b/vis_tracker.py
@@ -0,0 +1,104 @@
+# ---------------------------------------------------------
+# IOU Tracker
+# Copyright (c) 2019 TU Berlin, Communication Systems Group
+# Licensed under The MIT License [see LICENSE for details]
+# Written by Erik Bochinski
+# ---------------------------------------------------------
+
+import cv2
+
+KCF2_available = True
+try:
+ import KCF
+except ImportError:
+ KCF = None
+ KCF2_available = False
+
+
+class VisTracker:
+ kcf2_warning_printed = False
+
+ def __init__(self, tracker_type, bbox, img, keep_height_ratio=1.):
+ """ Wrapper class for various visual trackers."
+ Args:
+ tracker_type (str): name of the tracker. either the ones provided by opencv-contrib or KCF2 for a different
+ implementation for KCF (requires https://github.com/uoip/KCFcpp-py-wrapper)
+ bbox (tuple): box to initialize the tracker (x1, y1, x2, y2)
+ img (numpy.ndarray): image to intialize the tracker
+ keep_height_ratio (float, optional): float between 0.0 and 1.0 that determines the ratio of height of the
+ object to track to the total height of the object for visual tracking.
+ """
+ if tracker_type == 'KCF2' and not KCF:
+ tracker_type = 'KCF'
+ if not VisTracker.kcf2_warning_printed:
+ print("[warning] KCF2 not available, falling back to KCF. please see README.md for further details")
+ VisTracker.kcf2_warning_printed = True
+
+ self.tracker_type = tracker_type
+ self.keep_height_ratio = keep_height_ratio
+
+ if tracker_type == 'BOOSTING':
+ self.vis_tracker = cv2.TrackerBoosting_create()
+ elif tracker_type == 'MIL':
+ self.vis_tracker = cv2.TrackerMIL_create()
+ elif tracker_type == 'KCF':
+ self.vis_tracker = cv2.TrackerKCF_create()
+ elif tracker_type == 'KCF2':
+ self.vis_tracker = KCF.kcftracker(False, True, False, False) # hog, fixed_window, multiscale, lab
+ elif tracker_type == 'TLD':
+ self.vis_tracker = cv2.TrackerTLD_create()
+ elif tracker_type == 'MEDIANFLOW':
+ self.vis_tracker = cv2.TrackerMedianFlow_create()
+ elif tracker_type == 'GOTURN':
+ self.vis_tracker = cv2.TrackerGOTURN_create()
+ elif tracker_type == 'NONE': # dummy tracker that does nothing but fail
+ self.vis_tracker = None
+ self.ok = False
+ return
+ else:
+ raise ValueError("Unknown tracker type '{}".format(tracker_type))
+
+ y_max = img.shape[0] - 1
+ x_max = img.shape[1] - 1
+ #
+ bbox = list(bbox)
+ bbox[0] = max(0, min(bbox[0], x_max))
+ bbox[2] = max(0, min(bbox[2], x_max))
+ bbox[1] = max(0, min(bbox[1], y_max))
+ bbox[3] = max(0, min(bbox[3], y_max))
+
+ bbox = [bbox[0], bbox[1], bbox[2] - bbox[0], bbox[3] - bbox[1]] # x1, y1, x2, y2 -> x1, y1, w, h
+ bbox[3] *= self.keep_height_ratio
+
+ if self.tracker_type == 'KCF2':
+ self.vis_tracker.init(bbox, img)
+ self.ok = True
+ else:
+ self.ok = self.vis_tracker.init(img, tuple(bbox))
+ pass
+
+ def update(self, img):
+ """
+ Args:
+ img (numpy.ndarray): image for update
+
+ Returns:
+ bool: True if the update was successful, False otherwise
+ tuple: updated bounding box in (x1, y1, x2, y2) format
+ """
+ if not self.ok:
+ return False, [0, 0, 0, 0]
+
+ if self.tracker_type == 'KCF2':
+ bbox = self.vis_tracker.update(img)
+ ok = True
+ else:
+ ok, bbox = self.vis_tracker.update(img)
+ bbox = list(bbox)
+
+ bbox[3] /= self.keep_height_ratio
+ # x1, y1, w, h -> x1, y1, x2, y2
+ bbox[2] += bbox[0]
+ bbox[3] += bbox[1]
+
+ return ok, tuple(bbox)
diff --git a/visdrone-mot.sh b/visdrone-mot.sh
new file mode 100755
index 0000000..2d241aa
--- /dev/null
+++ b/visdrone-mot.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+# set these variables according to your setup
+visdrone_dir=/path/to/VisDrone2018-MOT-val # base directory of the split (VisDrone2018-MOT-val, VisDrone2018-MOT-train etc.)
+results_dir=results/VisDrone2018-MOT-val # output directory, will be created if not existing
+vis_tracker=MEDIANFLOW # [MEDIANFLOW, KCF2, NONE] parameter set as used in the paper
+
+
+if [ "${vis_tracker}" = "MEDIANFLOW" ]; then
+ options="-v MEDIANFLOW -sl 0.9 -sh 0.98 -si 0.1 -tm 23 --ttl 8 --nms 0.6 -fmt visdrone"
+elif [ "${vis_tracker}" = "KCF2" ]; then
+ options="-v KCF2 -sl 0.9 -sh 0.98 -si 0.05 -tm 23 --ttl 8 --nms 0.6 -fmt visdrone"
+elif [ "${vis_tracker}" = "NONE" ]; then
+ options="-sl 0.5 -sh 0.98 -si 0.05 -tm 7 --nms 0.6 -fmt visdrone"
+else
+ echo "unknown tracker '${vis_tracker}'"
+ exit
+fi
+
+mkdir -p ${results_dir}
+
+seq_dir=${visdrone_dir}/sequences
+
+echo "using '${vis_tracker}' option for visual tracking:"
+echo "${options}"
+for seq in $(ls $seq_dir); do
+ echo "processing ${seq} ...."
+
+ python demo.py -f ${seq_dir}/${seq}/{:07d}.jpg -d ${visdrone_dir}/detections/${seq}.txt \
+ -o ${results_dir}/${seq}.txt ${options}
+done