DeepLabCut/examples/testscript_openfielddata_augmentationcomparison.py at master · ebatty/DeepLabCut · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""

This is a test script to compare the loaders. tensorpack allows much more choices for augmentation. The parameters
can be set in pose_dataset_tensorpack.py and of course specifically in each pose_config.yaml file before training. In fact,
pose_dataset_tensorpack.py will fall back to default parameters if they are not defined in pose_config.yaml and one is
using dataset_type:'tensorpack'

This script creates one identical split for the openfield test dataset and trains it with the
standard loader and the tensorpack loader for k iterations in DLC 2.0 docker with TF 1.8 on a NVIDIA GTX 1080Ti.

My results were (Run with DLC 2.0.9 in Sept 2019)

**With standard loader:**

Training iterations:	%Training dataset	Shuffle number	 Train error(px)	 Test error(px)	p-cutoff used	Train error with p-cutoff	Test error with p-cutoff
10000	80	2	2.64	3.11	0.4	2.64	3.11
20000	80	2	2.26	2.72	0.4	2.26	2.72
30000	80	2	1.71	2.28	0.4	1.71	2.28
40000	80	2	1.88	2.61	0.4	1.88	2.61
50000	80	2	1.86	2.32	0.4	1.86	2.32
60000	80	2	1.92	2.42	0.4	1.92	2.42
70000	80	2	2.38	3.04	0.4	2.38	3.04
80000	80	2	1.55	2.34	0.4	1.55	2.34
90000	80	2	1.5	2.27	0.4	1.5	2.27
100000	80	2	1.52	2.34	0.4	1.52	2.34


**With tensorpack loader:**

Training iterations:	%Training dataset	Shuffle number	 Train error(px)	 Test error(px)	p-cutoff used	Train error with p-cutoff	Test error with p-cutoff
10000	80	3	2.35	2.91	0.4	2.35	2.91
20000	80	3	3.28	3.51	0.4	3.28	3.51
30000	80	3	1.57	2.24	0.4	1.57	2.24
40000	80	3	3.54	4.17	0.4	3.54	4.17
50000	80	3	1.76	2.74	0.4	1.76	2.74
60000	80	3	2.85	3.39	0.4	2.85	3.39
70000	80	3	3.88	4.71	0.4	3.88	4.71
80000	80	3	1.2	2.06	0.4	1.2	2.06
90000	80	3	2.2	3.07	0.4	2.2	3.07
100000	80	3	1.06	1.96	0.4	1.06	1.96


For details on TensorPack check out:

A Neural Net Training Interface on TensorFlow, with focus on speed + flexibility
https://github.com/tensorpack/tensorpack

My results were (Run with DLC 2.2b5 in May 2020) for 20k iterations

Imagaug augmentation:

Results for 20000  training iterations: 95 1 train error: 3.25 pixels. Test error: 4.98  pixels.
With pcutoff of 0.4  train error: 3.25 pixels. Test error: 4.98 pixels

Default augmentation:

Results for 20000  training iterations: 95 2 train error: 2.5 pixels. Test error: 4.08  pixels.
With pcutoff of 0.4  train error: 2.5 pixels. Test error: 4.08 pixels

Tensorpack augmentation:

Results for 20000  training iterations: 95 3 train error: 3.06 pixels. Test error: 4.78  pixels.
With pcutoff of 0.4  train error: 3.06 pixels. Test error: 4.78 pixels

Notice: despite the higher RMSE for imgaug due to the augmentation,
the network performs much better on the testvideo.

"""


import os
from pathlib import Path

os.environ["DLClight"] = "True"

import deeplabcut

# Loading example data set
path_config_file = os.path.join(os.getcwd(), "openfield-Pranav-2018-10-30/config.yaml")
cfg = deeplabcut.auxiliaryfunctions.read_config(path_config_file)


deeplabcut.load_demo_data(path_config_file)
maxiters = 20000

##create one split and make Shuffle 2 and 3 have the same split.
"""
trainIndices, testIndices=deeplabcut.mergeandsplit(path_config_file,trainindex=0,uniform=True)
deeplabcut.create_training_dataset(path_config_file,Shuffles=[2],trainIndices=trainIndices,testIndices=testIndices)
deeplabcut.create_training_dataset(path_config_file,Shuffles=[3],trainIndices=trainIndices,testIndices=testIndices)
for shuffle in [2,3]:
	if shuffle==3:
		posefile=os.path.join(cfg['project_path'],'dlc-models/iteration-'+str(cfg['iteration'])+'/'+ cfg['Task'] + cfg['date'] + '-trainset' + str(int(cfg['TrainingFraction'][0] * 100)) + 'shuffle' + str(shuffle),'train/pose_cfg.yaml')

		DLC_config=deeplabcut.auxiliaryfunctions.read_plainconfig(posefile)
		DLC_config['dataset_type']='tensorpack'
		deeplabcut.auxiliaryfunctions.write_plainconfig(posefile,DLC_config)
"""

###Note that the new function in DLC 2.1 simplifies network/augmentation comparisons greatly:
deeplabcut.create_training_model_comparison(
    path_config_file,
    num_shuffles=1,
    net_types=["resnet_50"],
    augmenter_types=["imgaug", "default", "tensorpack"],
)
for shuffle in [1, 2, 3]:
    if shuffle < 3:
        posefile, _, _ = deeplabcut.return_train_network_path(
            path_config_file, shuffle=shuffle
        )
        if shuffle == 2:  # Tensorpack:
            edits = {"rotate_max_deg_abs": 180, "noise_sigma": 0.01}
        elif shuffle == 1:  # imgaug
            edits = {"rotation": 180, "motion_blur": True}

        DLC_config = deeplabcut.auxiliaryfunctions.edit_config(posefile, edits)

    print("TRAIN NETWORK", shuffle)
    deeplabcut.train_network(
        path_config_file,
        shuffle=shuffle,
        saveiters=10000,
        displayiters=200,
        maxiters=maxiters,
        max_snapshots_to_keep=11,
    )

    print("EVALUATE")
    deeplabcut.evaluate_network(path_config_file, Shuffles=[shuffle], plotting=True)

    print("Analyze Video")

    videofile_path = os.path.join(
        os.getcwd(), "openfield-Pranav-2018-10-30", "videos", "m3v1mp4.mp4"
    )

    deeplabcut.analyze_videos(path_config_file, [videofile_path], shuffle=shuffle)

    print("Create Labeled Video and plot")
    deeplabcut.create_labeled_video(path_config_file, [videofile_path], shuffle=shuffle)
    deeplabcut.plot_trajectories(path_config_file, [videofile_path], shuffle=shuffle)