Full MOT17 benchmark & Hungarian algorithm

[akretz]

Hello everyone,

currently, there's only one MOT17 sequence used for benchmarking purposes. I've added a bash script which downloads the labels of all MOT17 sequences, unpacks them in the correct directory, runs the tracker on all sequences and finally evaluates all results. Also, I've removed short trajectories consisting of less than 10 frames from the output, because a lot of detection boxes are false positives and this should remove most of them.

I've also added an alternative matching algorithm using the Hungarian algorithm (also called Munkres assignment algorithm) instead of k-d trees. That algorithm, I thought, should perform better because it finds the minimum cost matching while the k-d tree algorithm only approximates a good solution. Running the MOT17 benchmark, I've found that the k-d tree algorithm has the following results

                IDF1   IDP   IDR  Rcll  Prcn   GT  MT  PT  ML    FP     FN  IDs    FM  MOTA  MOTP  IDt  IDa IDm
MOT17-04-SDP   62.2% 74.8% 53.2% 69.0% 97.1%   83  32  39  12   982  14737  121   367 66.7% 0.152   82   46  12
MOT17-04-DPM   29.4% 36.6% 24.5% 42.5% 63.5%   83   8  43  32 11616  27366  264   469 17.5% 0.224  110  151  12
MOT17-05-DPM   34.3% 48.2% 26.7% 40.2% 72.8%  133   9  63  61  1040   4133   94   150 23.9% 0.257  101   27  39
MOT17-05-FRCNN 44.8% 61.3% 35.3% 51.6% 89.6%  133  24  56  53   414   3347   63    69 44.7% 0.172   82   20  39
MOT17-11-FRCNN 55.4% 74.2% 44.2% 55.3% 92.8%   75  15  34  26   405   4219   45    50 50.5% 0.096   35   22  12
MOT17-05-SDP   43.1% 53.9% 35.9% 58.0% 87.0%  133  31  64  38   597   2905   84   128 48.2% 0.165  108   21  45
MOT17-02-FRCNN 31.6% 52.8% 22.6% 34.6% 81.1%   62   6  25  31  1502  12148   93   127 26.0% 0.126   55   49  11
MOT17-09-FRCNN 43.6% 61.2% 33.8% 53.4% 96.8%   26   6  16   4    95   2481   37    39 50.9% 0.096   27   16   6
MOT17-10-FRCNN 40.0% 46.8% 35.0% 57.9% 77.6%   57  15  35   7  2150   5405  228   311 39.4% 0.162  147   89  16
MOT17-04-FRCNN 49.5% 67.8% 39.0% 53.2% 92.4%   83  17  43  23  2077  22271   93    90 48.6% 0.108   42   56   5
MOT17-13-DPM   28.8% 45.8% 21.0% 27.1% 59.2%  110  10  42  58  2172   8489   75   207  7.8% 0.282   48   42  19
MOT17-13-SDP   60.3% 78.0% 49.1% 54.5% 86.6%  110  43  25  42   980   5294   72   199 45.5% 0.212   70   24  27
MOT17-02-SDP   34.3% 46.8% 27.1% 44.7% 77.1%   62   9  34  19  2468  10275  176   353 30.5% 0.201  114   72  13
MOT17-13-FRCNN 50.4% 61.1% 42.9% 55.9% 79.6%  110  35  46  29  1667   5129  200   259 39.9% 0.169  141   83  34
MOT17-02-DPM   23.3% 44.5% 15.8% 25.7% 72.4%   62   4  19  39  1814  13811  107   218 15.3% 0.260   62   55  12
MOT17-10-SDP   48.9% 56.9% 42.8% 67.4% 89.5%   57  23  30   4  1010   4189  163   290 58.2% 0.205  108   60   9
MOT17-09-DPM   35.1% 35.8% 34.4% 57.6% 59.9%   26   5  19   2  2055   2260   77   161 17.5% 0.275   49   32   7
MOT17-09-SDP   48.5% 63.5% 39.2% 57.6% 93.2%   26   7  17   2   222   2260   46    78 52.5% 0.150   35   18   7
MOT17-10-DPM   30.9% 39.6% 25.3% 41.7% 65.3%   57   7  30  20  2852   7479  119   299 18.6% 0.264   61   66  10
MOT17-11-DPM   46.5% 54.9% 40.3% 53.2% 72.5%   75  10  30  35  1904   4413   52    68 32.5% 0.224   27   31   9
MOT17-11-SDP   53.2% 62.9% 46.2% 66.5% 90.7%   75  19  39  17   646   3161   55   100 59.1% 0.149   40   28  15
OVERALL        43.5% 56.7% 35.3% 50.8% 81.6% 1638 335 749 554 38668 165772 2264  4032 38.6% 0.174 1544 1008 359

while the Hungarian algorithm has the following results

                IDF1   IDP   IDR  Rcll  Prcn   GT  MT  PT  ML    FP     FN  IDs    FM  MOTA  MOTP  IDt IDa IDm
MOT17-04-SDP   66.0% 77.0% 57.8% 73.2% 97.4%   83  38  33  12   912  12730  119   340 71.1% 0.153   92  28  14
MOT17-04-DPM   29.4% 33.6% 26.2% 47.1% 60.4%   83  10  44  29 14711  25159  499   715 15.1% 0.227  348 111  16
MOT17-05-DPM   31.6% 41.6% 25.4% 43.3% 71.0%  133  11  64  58  1225   3920  135   168 23.7% 0.256  142  25  47
MOT17-05-FRCNN 48.7% 66.0% 38.6% 53.0% 90.5%  133  23  61  49   383   3251   61    79 46.6% 0.171   82  24  45
MOT17-11-FRCNN 47.1% 62.7% 37.7% 55.7% 92.6%   75  15  33  27   420   4183   52    50 50.7% 0.095   41  22  13
MOT17-05-SDP   45.8% 54.6% 39.5% 63.7% 88.1%  133  33  70  30   598   2509  100   137 53.6% 0.165  128  24  58
MOT17-02-FRCNN 33.1% 54.9% 23.7% 34.9% 80.8%   62   6  27  29  1545  12097   97   139 26.1% 0.125   70  41  14
MOT17-09-FRCNN 49.5% 67.6% 39.1% 57.1% 98.8%   26   5  19   2    36   2284   34    43 55.8% 0.102   29  12   7
MOT17-10-FRCNN 39.7% 45.6% 35.2% 59.8% 77.4%   57  14  38   5  2248   5160  249   330 40.4% 0.165  159  89  19
MOT17-04-FRCNN 52.0% 70.5% 41.1% 54.2% 92.9%   83  18  42  23  1975  21774   87    95 49.9% 0.108   49  42   4
MOT17-13-DPM   28.9% 44.6% 21.4% 28.0% 58.3%  110  11  42  57  2333   8378   73   206  7.4% 0.281   43  42  16
MOT17-13-SDP   60.9% 76.9% 50.4% 56.9% 86.7%  110  46  25  39  1015   5021   82   226 47.4% 0.214   84  19  26
MOT17-02-SDP   29.8% 38.8% 24.2% 47.8% 76.8%   62  11  36  15  2692   9690  260   391 32.0% 0.201  191  71  21
MOT17-13-FRCNN 47.4% 54.3% 42.1% 59.2% 76.3%  110  38  46  26  2147   4746  276   330 38.4% 0.177  195  89  41
MOT17-02-DPM   21.7% 38.5% 15.1% 27.3% 69.6%   62   6  19  37  2211  13515  155   221 14.5% 0.263  102  57  14
MOT17-10-SDP   50.6% 56.5% 45.9% 72.2% 89.0%   57  30  23   4  1147   3564  187   306 61.9% 0.206  129  46  10
MOT17-09-DPM   37.2% 36.0% 38.5% 61.8% 57.8%   26   6  18   2  2402   2034  119   156 14.5% 0.269   87  27  11
MOT17-09-SDP   41.9% 52.0% 35.1% 64.3% 95.4%   26   9  16   1   167   1899   77    92 59.8% 0.154   72  13  11
MOT17-10-DPM   30.5% 37.6% 25.7% 43.5% 63.8%   57   9  27  21  3173   7251  141   300 17.7% 0.263   74  66  13
MOT17-11-DPM   34.1% 37.4% 31.3% 56.3% 67.3%   75  12  29  34  2585   4119  119    97 27.7% 0.223   86  41  19
MOT17-11-SDP   56.7% 64.5% 50.7% 71.1% 90.4%   75  26  33  16   710   2727   82   107 62.7% 0.151   62  27  19
OVERALL        43.6% 54.4% 36.4% 53.7% 80.2% 1638 377 745 516 44635 156011 3004  4528 39.6% 0.176 2265 916 438

Since the Hungarian algorithm finds optimal matchings, it matches tracked objects to detection boxes more often, which results in fewer false negatives (FN), but also more false positives (FP). Also, we have more mostly tracked (MT, trajectories which are tracked for at least 80% of frames) trajectories.

In general, you can see that the tracker using the Hungarian algorithm performs slightly worse when the detector has a lot of false positives (compare the results which use the DPM detector) and slightly better when the detector doesn't have that many false positives. I suspect the Hungarian algorithm should be slightly better when we use YOLO detections, as the number of false positives tends to be rather small.

I've added a parameter which can switch between the k-d tree algorithm and the Hungarian algorithm and left the k-d tree algorithm as default. In order to evaluate which algorithm is a better fit for OpenDataCam, I guess further evaluation needs to be done. Using YOLO detections instead of detections given by the MOT17 benchmark for evaluation purposes is probably a good idea. Since traffic scenarios seem to be a major use case of OpenDataCam, using the UA-DETRAC dataset for future evaluations might be a good idea.

[akretz]

If you ignore DPM detection with negative confidence, a lot of false positives get removed and the Hungarian algorithm performs better there than the k-d tree algorithm. Also, an IoU threshold of 0.05 seems really low to me. That results in a lot of bad matches with very small overlap and a lot of valid detections falsely removed as double detections. If you also change that to 0.2, you get the following result for the Hungarian algorithm:

                IDF1   IDP   IDR  Rcll  Prcn   GT  MT  PT  ML    FP     FN  IDs    FM  MOTA  MOTP  IDt  IDa IDm
MOT17-04-SDP   68.9% 78.0% 61.7% 77.0% 97.4%   83  47  26  10   974  10915  138   321 74.7% 0.154   81   51  12
MOT17-04-DPM   37.7% 56.4% 28.3% 43.5% 86.7%   83  10  40  33  3172  26865  208   354 36.4% 0.219   66  143   7
MOT17-05-DPM   39.8% 63.6% 28.9% 38.9% 85.5%  133  12  53  68   455   4228   66   112 31.3% 0.249   51   31  16
MOT17-05-FRCNN 55.8% 72.1% 45.5% 56.7% 89.9%  133  30  58  45   439   2997   58    71 49.5% 0.172   52   30  24
MOT17-11-FRCNN 53.4% 66.7% 44.6% 62.0% 92.8%   75  20  32  23   456   3586   49    51 56.6% 0.097   20   34   6
MOT17-05-SDP   55.6% 63.9% 49.3% 67.5% 87.5%  133  41  64  28   666   2249   89   120 56.6% 0.169   79   42  35
MOT17-02-FRCNN 34.6% 54.6% 25.3% 36.6% 78.9%   62   7  27  28  1816  11778  103   145 26.3% 0.130   45   68  13
MOT17-09-FRCNN 50.0% 66.6% 40.1% 59.3% 98.6%   26   7  17   2    45   2167   31    39 57.9% 0.107   17   21   7
MOT17-10-FRCNN 41.0% 45.1% 37.6% 61.9% 74.4%   57  16  36   5  2737   4889  293   342 38.3% 0.168  160  139  17
MOT17-04-FRCNN 55.8% 73.9% 44.9% 56.6% 93.1%   83  20  40  23  1995  20662   77    97 52.2% 0.109   12   68   3
MOT17-13-DPM   26.7% 72.6% 16.4% 19.9% 88.3%  110   9  27  74   308   9322   34    83 17.0% 0.272   11   30   7
MOT17-13-SDP   57.6% 70.9% 48.6% 58.1% 84.8%  110  47  27  36  1210   4875  116   194 46.7% 0.216   60   38  24
MOT17-02-SDP   39.3% 49.0% 32.9% 51.3% 76.4%   62  12  38  12  2944   9045  210   365 34.3% 0.203  105  105  12
MOT17-13-FRCNN 46.4% 51.5% 42.3% 61.0% 74.3%  110  37  50  23  2455   4538  316   326 37.2% 0.179  189  136  26
MOT17-02-DPM   20.9% 56.4% 12.8% 19.9% 87.7%   62   5  13  44   520  14875   51   105 16.9% 0.251   18   35   2
MOT17-10-SDP   48.9% 52.9% 45.4% 75.0% 87.3%   57  32  21   4  1401   3212  211   322 62.4% 0.209  118   84   8
MOT17-09-DPM   39.1% 45.2% 34.4% 60.5% 79.5%   26   4  18   4   831   2105   77   130 43.4% 0.272   47   30   2
MOT17-09-SDP   51.4% 61.5% 44.2% 68.2% 94.9%   26  11  15   0   196   1692   58    91 63.5% 0.153   40   24   8
MOT17-10-DPM   32.8% 54.1% 23.5% 37.9% 87.1%   57   7  18  32   724   7972   81   138 31.6% 0.251   24   60   6
MOT17-11-DPM   46.8% 62.7% 37.3% 55.6% 93.3%   75  12  26  37   375   4193   44    64 51.1% 0.220   13   34   5
MOT17-11-SDP   60.2% 65.8% 55.4% 76.0% 90.2%   75  30  33  12   780   2263   62   104 67.1% 0.153   26   41  13
OVERALL        48.3% 63.5% 39.0% 54.2% 88.2% 1638 416 679 543 24499 154428 2372  3574 46.2% 0.173 1234 1244 253

[tdurand]

@akretz wanted to do this for ages, thanks ! I'm super busy this end of week but next week I'll have a look

[vsaw]

@akretz Thank you for the contributions. Better late then never I guess ;-)