mdgru.eval package

Submodules

mdgru.eval.tf module

class mdgru.eval.tf.SupervisedEvaluationTensorflow(modelcls, datacls, kw)

Bases: mdgru.eval.SupervisedEvaluation

Evaluation class for tensorflow backend

Parameters:

• kw (dict containing the following options.) –
  • use_tensorboard [default: True] Dont use tensorboard
  • image_summaries_each [default: 100] Store image summaries in tensorboard every # iterations
  • restore_optimistically [default: False]
  • only_cpu [default: False] Only use cpu
  • gpubound [default: 1.0] manage how much of the memory of the gpu can be used
• modelcls (cls) – Python class defining the model to be evaluated
• datacls (cls) – Python class defining data loading and saving

_defaults = {'gpubound': {'value': 1.0, 'name': 'gpu_bound_fraction', 'help': 'manage how much of the memory of the gpu can be used', 'type': <class 'float'>}, 'image_summaries_each': {'value': 100, 'help': 'Store image summaries in tensorboard every # iterations'}, 'only_cpu': {'value': False, 'help': 'Only use cpu'}, 'restore_optimistically': False, 'use_tensorboard': {'value': True, 'help': 'Dont use tensorboard', 'invert_meaning': 'dont_'}}

_load(f)

_optimistic_restore(session, save_file)

_predict(batch, dropout, testing)

_predict_with_loss(batch, batchlabs)

_save(f)

_set_session(sess, cachefolder)

_train(batch, batchlabs)

add_summary_simple_value(text, value)

get_globalstep()

load(f)

loads model at location f from disk

Parameters:f (str) – location of stored model

save(f)

saves model to disk at location f

Parameters:f (str) – location to save model to

set_session(sess, cachefolder, train=False)

test_all_available(batch_size=None, dc=None, return_results=False, dropout=None, testing=False)

Completely evaluates each full image in tps using grid sampling.

Parameters:

• batch_size (int) – minibatch size to compute on
• dc (datacollection instance, optional) – datacollection to sample from
• return_results (bool) – should results be returned or stored right away?
• dropout (float) – keeprate of dropconnect for inference
• testing –

Returns:

either tuple of predictions and errors or only errors, depending on return_results flag

test_all_random(batch_size=None, dc=None, resample=True)

Test random samples

Parameters:

• batch_size (int) – minibatch size to compute on
• dc (datacollection instance, optional) – datacollection to sample from
• resample (bool) – indicates if we need to sample before evaluating

Returns:

tuple of loss and prediction ndarray

test_scores(pred, ref)

Evaluates all selected scores between reference data ref and prediction pred.

Parameters:

• pred (ndarray) – prediction, as probability distributions per pixel / voxel
• ref (ndarray) – labelmap, either as probability distributions per pixel / voxel or as label map

train()

Measures and logs time when performing data sampling and training iteration.

mdgru.eval.torch module

class mdgru.eval.torch.SupervisedEvaluationTorch(modelcls, datacls, kw)

Bases: mdgru.eval.SupervisedEvaluation

Evaluation class for the pytorch backend

Parameters:

• kw (dict containing the following options.) –
• modelcls (cls) – Python class defining the model to be evaluated
• datacls (cls) – Python class defining the loading and saving of the data being evaluated here

_defaults = {}

_load(f)

Load model

_predict(batch, dropout, testing)

predict given our graph for batch. Be careful as this method returns results always in NHWC or NDHWC

_predict_with_loss(batch, batchlabs)

run evaluation and calculate loss

_save(f)

Save model

_set_session(sess, cachefolder)

_train(batch, batchlabs)

set inputs and run torch training iteration

add_summary_simple_value(text, value)

check_input(batch, batchlabs=None)

Method to check correctness of input and convert them to cuda pytorch tensors

Parameters:

• batch (ndarray) – input data to be moved to pytorch
• batchlabs (ndarray) – label information to be moved to pytorch

get_globalstep()

load(f)

loads model at location f from disk

Parameters:f (str) – location of stored model

save(f)

saves model to disk at location f

Parameters:f (str) – location to save model to

set_session(sess, cachefolder, train=False)

test_all_available(batch_size=None, dc=None, return_results=False, dropout=None, testing=False)

Completely evaluates each full image in tps using grid sampling.

Parameters:

• batch_size (int) – minibatch size to compute on
• dc (datacollection instance, optional) – datacollection to sample from
• return_results (bool) – should results be returned or stored right away?
• dropout (float) – keeprate of dropconnect for inference
• testing –

Returns:

either tuple of predictions and errors or only errors, depending on return_results flag

test_all_random(batch_size=None, dc=None, resample=True)

Test random samples

Parameters:

• batch_size (int) – minibatch size to compute on
• dc (datacollection instance, optional) – datacollection to sample from
• resample (bool) – indicates if we need to sample before evaluating

Returns:

tuple of loss and prediction ndarray

test_scores(pred, ref)

Evaluates all selected scores between reference data ref and prediction pred.

Parameters:

• pred (ndarray) – prediction, as probability distributions per pixel / voxel
• ref (ndarray) – labelmap, either as probability distributions per pixel / voxel or as label map

train()

Measures and logs time when performing data sampling and training iteration.

Module contents

class mdgru.eval.SupervisedEvaluation(modelcls, datacls, kw)

Bases: object

Handler for the evaluation of model defined in modelcls using data coming from datacls.

Parameters:

• kw (dict containing the following options.) –
  • dropout_rate [default: 0.5] “keep rate” for weights using dropconnect. The higher the value, the closer the sampled models to the full model.
  • namespace [default: default] override default model name (if no ckpt is provided). Probably not a good idea!
  • only_save_labels [default: False] save only labels and no probability distributions
  • validate_same [default: True] always pick other random samples for validation!
  • evaluate_uncertainty_times [default: 1] Number times we want to evaluate one volume. This only makes sense using a keep rate of less than 1 during evaluation (dropout_during_evaluation less than 1)
  • evaluate_uncertainty_dropout [default: 1.0] Keeprate of weights during evaluation. Useful to visualize uncertainty in conjunction with a number of samples per volume
  • evaluate_uncertainty_saveall [default: False] Save each evaluation sample per volume. Without this flag, only the standard deviation and mean over all samples is kept.
  • show_f05 [default: True]
  • show_f1 [default: True]
  • show_f2 [default: True]
  • show_l2 [default: True]
  • show_cross_entropy [default: True]
  • print_each [default: 1] print execution time and losses each # iterations
  • batch_size [default: 1] Minibatchsize
  • datapath path where training, validation and testing folders lie. Can also be some other path, as long as the other locations are provided as absolute paths. An experimentsfolder will be created in this folder, where all runs and checkpoint files will be saved.
  • locationtraining [default: None] absolute or relative path to datapath to the training data. Either a list of paths to the sample folders or one path to a folder where samples should be automatically determined.
  • locationtesting [default: None] absolute or relative path to datapath to the testing data. Either a list of paths to the sample folders or one path to a folder where samples should be automatically determined.
  • locationvalidation [default: None] absolute or relative path to datapath to the validation data. Either a list of paths to the sample folders or one path to a folder where samples should be automatically determined.
  • output_dims number of output channels, e.g. number of classes the model needs to create a probability distribution over.
  • windowsize window size to be used during training, validation and testing, if not specified otherwise
  • padding [default: [0]] padding to be used during training, validation and testing, if not specified otherwise. During training, the padding specifies the amount a patch is allowed to reach outside of the image along all dimensions, during testing, it specifies also the amount of overlap needed between patches.
  • windowsizetesting [default: None] override windowsize for testing
  • windowsizevalidation [default: None]
  • paddingtesting [default: None] override padding for testing
  • paddingvalidation [default: None]
  • testbatchsize [default: 1] batchsize for testing
• modelcls (cls) – Python class defining the model to evaluate
• datacls (cls) – Python class implementing the data loading and storing

_defaults = {'batch_size': {'value': 1, 'help': 'Minibatchsize', 'type': <class 'int'>, 'name': 'batchsize', 'short': 'b'}, 'datapath': {'help': 'path where training, validation and testing folders lie. Can also be some other path, as long as the other locations are provided as absolute paths. An experimentsfolder will be created in this folder, where all runs and checkpoint files will be saved.'}, 'dropout_rate': {'value': 0.5, 'help': '"keep rate" for weights using dropconnect. The higher the value, the closer the sampled models to the full model.'}, 'evaluate_uncertainty_dropout': {'value': 1.0, 'type': <class 'float'>, 'help': 'Keeprate of weights during evaluation. Useful to visualize uncertainty in conjunction with a number of samples per volume', 'name': 'dropout_during_evaluation'}, 'evaluate_uncertainty_saveall': {'value': False, 'help': 'Save each evaluation sample per volume. Without this flag, only the standard deviation and mean over all samples is kept.', 'name': 'save_individual_evaluations'}, 'evaluate_uncertainty_times': {'value': 1, 'type': <class 'int'>, 'help': 'Number times we want to evaluate one volume. This only makes sense using a keep rate of less than 1 during evaluation (dropout_during_evaluation less than 1)', 'name': 'number_of_evaluation_samples'}, 'locationtesting': {'value': None, 'help': 'absolute or relative path to datapath to the testing data. Either a list of paths to the sample folders or one path to a folder where samples should be automatically determined.', 'nargs': '+'}, 'locationtraining': {'value': None, 'help': 'absolute or relative path to datapath to the training data. Either a list of paths to the sample folders or one path to a folder where samples should be automatically determined.', 'nargs': '+'}, 'locationvalidation': {'value': None, 'help': 'absolute or relative path to datapath to the validation data. Either a list of paths to the sample folders or one path to a folder where samples should be automatically determined.', 'nargs': '+'}, 'namespace': {'value': 'default', 'help': 'override default model name (if no ckpt is provided). Probably not a good idea!', 'alt': ['modelname']}, 'only_save_labels': {'value': False, 'help': 'save only labels and no probability distributions'}, 'output_dims': {'help': 'number of output channels, e.g. number of classes the model needs to create a probability distribution over.', 'type': <class 'int'>, 'alt': ['nclasses']}, 'padding': {'help': 'padding to be used during training, validation and testing, if not specified otherwise. During training, the padding specifies the amount a patch is allowed to reach outside of the image along all dimensions, during testing, it specifies also the amount of overlap needed between patches.', 'value': [0], 'nargs': '+', 'short': 'p', 'type': <class 'int'>}, 'paddingtesting': {'value': None, 'help': 'override padding for testing', 'nargs': '+', 'type': <class 'int'>}, 'paddingvalidation': None, 'print_each': {'value': 1, 'help': 'print execution time and losses each # iterations', 'type': <class 'int'>}, 'show_cross_entropy': True, 'show_f05': True, 'show_f1': True, 'show_f2': True, 'show_l2': True, 'testbatchsize': {'value': 1, 'help': 'batchsize for testing'}, 'validate_same': {'value': True, 'help': 'always pick other random samples for validation!', 'invert_meaning': 'dont_'}, 'windowsize': {'type': <class 'int'>, 'short': 'w', 'help': 'window size to be used during training, validation and testing, if not specified otherwise', 'nargs': '+'}, 'windowsizetesting': {'value': None, 'help': 'override windowsize for testing', 'nargs': '+', 'type': <class 'int'>}, 'windowsizevalidation': None}

_load(f)

Load model in current framework from f

Parameters:f (location of stored model) –

_predict(batch, dropout, testing)

Predict given batch and keeprate dropout.

Parameters:

• batch (ndarray) –
• dropout (float) – Keeprate for dropconnect
• testing –

Returns:

ndarray (Prediction based on data batch)

_predict_with_loss(batch, batchlabs)

Predict for given batch and return loss compared to labels in batchlabs

Parameters:

• batch (image data) –
• batchlabs (corresponding label data) –

Returns:

tuple of ndarray prediction and losses

_save(f)

Save to file f in current framework

Parameters:f (location to save model at) –

_set_session(sess, cachefolder)

_train()

Performs one training iteration in respective framework and returns loss(es)

add_summary_simple_value(text, value)

get_globalstep()

Return number of iterations this model has been trained in

Returns:int (iteration count)

load(f)

loads model at location f from disk

Parameters:f (str) – location of stored model

save(f)

saves model to disk at location f

Parameters:f (str) – location to save model to

set_session(sess, cachefolder, train=False)

test_all_available(batch_size=None, dc=None, return_results=False, dropout=None, testing=False)

Completely evaluates each full image in tps using grid sampling.

Parameters:

• batch_size (int) – minibatch size to compute on
• dc (datacollection instance, optional) – datacollection to sample from
• return_results (bool) – should results be returned or stored right away?
• dropout (float) – keeprate of dropconnect for inference
• testing –

Returns:

either tuple of predictions and errors or only errors, depending on return_results flag

test_all_random(batch_size=None, dc=None, resample=True)

Test random samples

Parameters:

• batch_size (int) – minibatch size to compute on
• dc (datacollection instance, optional) – datacollection to sample from
• resample (bool) – indicates if we need to sample before evaluating

Returns:

tuple of loss and prediction ndarray

test_scores(pred, ref)

Evaluates all selected scores between reference data ref and prediction pred.

Parameters:

• pred (ndarray) – prediction, as probability distributions per pixel / voxel
• ref (ndarray) – labelmap, either as probability distributions per pixel / voxel or as label map

train()

Measures and logs time when performing data sampling and training iteration.