cd.data
This submodule contains data related code and numpy operations.
Datasets
BBBC039
- class BBBC039Test(directory, download=False)
BBBC039 Test.
Test split of the BBBC039 dataset.
References
https://bbbc.broadinstitute.org/BBBC039
- Parameters:
directory – Root directory.
download – Whether to download the dataset.
- class BBBC039Train(directory, download=False)
BBBC039 Train.
Training split of the BBBC039 dataset.
References
https://bbbc.broadinstitute.org/BBBC039
- Parameters:
directory – Root directory.
download – Whether to download the dataset.
- class BBBC039Val(directory, download=False)
BBBC039 Validation.
Validation split of the BBBC039 dataset.
References
https://bbbc.broadinstitute.org/BBBC039
- Parameters:
directory – Root directory.
download – Whether to download the dataset.
- download_bbbc039(directory)
Download BBBC039.
Download and extract the BBBC039 dataset to given directory.
References
https://bbbc.broadinstitute.org/BBBC039
- Parameters:
directory – Root directory.
BBBC041
- class BBBC041Test(directory, download=False)
BBBC041 Test data.
References
https://bbbc.broadinstitute.org/BBBC041
- Parameters:
directory – Data directory.
download – Whether to download data.
- class BBBC041Train(directory, download=False)
BBBC041 Train data.
References
https://bbbc.broadinstitute.org/BBBC041
- Parameters:
directory – Data directory.
download – Whether to download data.
- download_bbbc041(directory, url='https://data.broadinstitute.org/bbbc/BBBC041/malaria.zip')
Download BBBC041.
Download and extract the BBBC041 dataset to given directory.
References
https://bbbc.broadinstitute.org/BBBC041
- Parameters:
directory – Root directory.
url – Download URL (this dataset is distributed in a single zip file).
Synth
- class SynthTest(directory, download=False, cache=True)
Synth Test data.
- Parameters:
directory – Data directory. E.g.
directory='data/synth'
.download – Whether to download data.
cache – Whether to hold data in memory.
- class SynthTrain(directory, download=False, cache=True)
Synth Train data.
- Parameters:
directory – Data directory. E.g.
directory='data/synth'
.download – Whether to download data.
cache – Whether to hold data in memory.
- class SynthVal(directory, download=False, cache=True)
Synth Val data.
- Parameters:
directory – Data directory. E.g.
directory='data/synth'
.download – Whether to download data.
cache – Whether to hold data in memory.
- download_synth(directory, url='https://celldetection.org/data/synth.zip')
Download Synth.
Download and extract the Synth dataset to given directory.
- Parameters:
directory – Root directory.
url – Download URL (this dataset is distributed in a single zip file).
Data Operations
- channels_first2channels_last(inputs: ndarray, spatial_dims=2, has_batch=False) ndarray
Channels first to channels last.
- Parameters:
inputs – Input array.
spatial_dims – Number of spatial dimensions.
has_batch – Whether inputs has a batch dimension.
- Returns:
Transposed array.
- channels_last2channels_first(inputs: ndarray, spatial_dims=2, has_batch=False) ndarray
Channels last to channels first.
- Parameters:
inputs – Input array.
spatial_dims – Number of spatial dimensions.
has_batch – Whether inputs has a batch dimension.
- Returns:
Transposed array.
- ensure_tensor(x, device=None, dtype=torch.float32)
Ensure tensor.
Mapping ndarrays to Tensor. Possible shape mappings: - (h, w) -> (1, 1, h, w) - (h, w, c) -> (1, c, h, w) - (b, c, h, w) -> (b, c, h, w)
- Parameters:
x – Inputs.
device – Either Device or a Module or Tensor to retrieve the device from.
dtype – Data type.
- Returns:
Tensor
- labels2crops(labels: ndarray, image: ndarray)
Labels to crops.
Crop all objects that are represented in
labels
from givenimage
and return a list of all image crops, and a list of masks, each marking object pixels for respective crop.- Parameters:
labels – Label image. Array[h, w(, c)].
image – Image. Array[h, w, …].
- Returns:
(crop_list, mask_list)
- labels2properties(labels: ndarray, *properties, iter_channels=True, **kwargs)
Labels to properties.
References
[1] https://scikit-image.org/docs/stable/api/skimage.measure.html#skimage.measure.regionprops
- Parameters:
labels – Label image.
*properties – Property names. See [1] for details.
iter_channels – Whether to iterate channel axis of label image. If False label image is processed as is.
**kwargs – Keyword arguments for skimage.measure.regionprops.
- Returns:
List of property lists.
- labels2property_table(labels: ndarray, *properties, iter_channels=True, **kwargs) DataFrame
Labels to property table.
References
[1] https://scikit-image.org/docs/stable/api/skimage.measure.html#skimage.measure.regionprops
- Parameters:
labels – Label image.
*properties – Property names. See [1] for details.
iter_channels – Whether to iterate channel axis of label image. If False label image is processed as is.
**kwargs – Keyword arguments for skimage.measure.regionprops_table.
- Returns:
Table (pd.DataFrame) of properties.
- normalize_percentile(image, percentile=99.9, to_uint8=True)
- pad_to_div(v, div=32, nd=2, **kwargs)
Pad to div.
Applies padding to input Array to make it divisible by div.
- Parameters:
v – Input array.
div – Div tuple. If single integer, nd is used to define number of dimensions to pad.
nd – Number of dimensions to pad. Only used if div is not a tuple or list.
**kwargs – Additional keyword arguments for np.pad.
- Returns:
Padded Array.
- pad_to_size(v, size, **kwargs)
Pad tp size.
Applies padding to end of each dimension.
- Parameters:
v – Input array.
size – Size tuple. First element corresponds to first dimension of input v.
**kwargs – Additional keyword arguments for np.pad.
- Returns:
Padded Array.
- padding_stack(*images, axis=0) ndarray
Padding stack.
Stack images along axis. If images have different shapes all images are padded to larges shape.
- Parameters:
*images – Images.
axis – Axis used for stacking.
- Returns:
Array
- random_crop(inputs, size=None, *args, return_coords=False, return_slices=False, **kwargs)
- regionprops2d(label_image, intensity_image=None, cache=True, *, extra_properties=None, spacing=None, offset=None)
Regionprops 2d.
Helper function that allows to use skimage.measure.regionprops with label images that have channels.
Note
Labels may not yield in order!
- Parameters:
label_image – Array[h, w] or Array[h, w, c].
intensity_image
cache
extra_properties
spacing
offset
Returns:
- resample_contours(contours, num=None, close=True, epsilon=1e-06)
Resample contour.
Sample ´´num´´ equidistant points on each contour in
contours
.Notes
Works for closed and open contours.
- Parameters:
contours – Contours to sample from. Array[…, num’, 2] or list of Arrays.
num – Number of points.
close – Set True if
contours
contains closed contours, with the end point not being equal to the start point. Set False otherwise.epsilon – Epsilon.
- Returns:
Array[…, num, 2] or list of Arrays.
- rgb_to_scalar(inputs: ndarray, dtype='int32')
RGB to scalar.
Convert RGB data to scalar, while maintaining color uniqueness.
- Parameters:
inputs – Input array. Shape ([d1, d2, …, dn,] 3)
dtype – Data type
- Returns:
Output array. Shape ([d1, d2, …, dn])
- rle2mask(code, shape, transpose=True, min_index=1, constant=1) ndarray
Run length encoding to mask.
Convert run length encoding to mask image.
- Parameters:
code – Run length code. As ndarray: array([idx0, len0, idx1, len1, …]) or array([[idx0, len0], [idx1, len1], …]) As list: [idx0, len0, idx1, len1, …] or [[idx0, len0], [idx1, len1], …] As str: ‘idx0 len0 idx1 len1 …’
shape – Mask shape.
transpose – If True decode row by row, otherwise decode column by column.
min_index – Smallest pixel index. Depends on rle encoding.
constant – Pixels marked by rle are set to this value.
- Returns:
Mask image.
- split(n: int, *splits, shuffle=True, seed=None)
Split.
Splits a range of indices into multiple sets based on the given fractions.
- Parameters:
n – The total number of indices.
*splits – Variable length list of floats representing the fraction of the dataset for each split.
shuffle – Whether to shuffle the indices before splitting.
seed – Seed for the random number generator.
- Returns:
Split indices.
- to_tensor(inputs: ndarray, spatial_dims=2, transpose=False, has_batch=False, dtype=None, device=None) Tensor
Array to Tensor.
Converts numpy array to Tensor and optionally transposes from channels last to channels first.
- Parameters:
inputs – Input array.
transpose – Whether to transpose channels from channels last to channels first.
spatial_dims – Number of spatial dimensions.
has_batch – Whether inputs has a batch dimension.
dtype – Data type of output Tensor.
device – Device of output Tensor.
- Returns:
Tensor.
- transpose_spatial(inputs: ndarray, inputs_channels_last=True, spatial_dims=2, has_batch=False)
- universal_dict_collate_fn(batch, check_padding=True) OrderedDict
- boxes2masks(boxes, size)
- fill_label_gaps_(labels)
Fill label gaps.
Ensure that labels greater zero are within interval [1, num_unique_labels_greater_zero]. Works fast if gaps are unlikely, slow otherwise. Alternatively consider using np.vectorize. Labels <= 0 are preserved as is.
- Parameters:
labels
Returns:
- fill_padding_(inputs, padding: int, constant=-1, preserve_existing=True, axes=(0, 1))
- filter_instances_(labels, partials=True, partials_border=1, min_area=4, max_area=None, constant=-1, continuous=True)
Filter instances from label image.
Note
Filtered instance labels are set to constant. Labels might not be continuous afterwards.
- Parameters:
labels
partials
partials_border
min_area
max_area
constant
continuous
Returns:
- relabel_(label_stack, axis=2)
Relabel.
Inplace relabeling of a label stack. After applying this op the labels in label_stack are continuous, starting at 1. Negative labels remain untouched.
Notes
Uses label function from sklearn.morphology
- Parameters:
label_stack – Array[height, width, channels].
axis – Channel axis.
- remove_padding(inputs, padding: int)
- remove_partials_(label_stack, border=1, constant=-1)
- stack_labels(*maps, axis=2, dtype='int32', relabel=True)
Stack labels.
- Parameters:
*maps – List[Union[Array[h, w], Array[h, w, 3]]. Grayscale or rgb label maps. Rgb labels are assumed to encode labels with color and are converted to grayscale labels before stacking.
axis – Stacking axis.
dtype – Output data type.
relabel – Whether to assign new labels or not.
- Returns:
Array[h, w, c]. Label image.
- unary_masks2labels(unary_masks, transpose=True)
Unary masks to labels.
- Parameters:
unary_masks – List[Array[height, width]] or Array[num_objects, height, width] List of masks. Each mask is assumed to contain exactly one object.
transpose – If True label images are in channels last format, otherwise channels first.
- Returns:
Label image. Array[height, width, num_objects] if transpose else Array[num_objects, height, width].
CPN Operations
- class CPNTargetGenerator(samples, order, random_sampling=True, remove_partials=False, min_fg_dist=0.75, max_bg_dist=0.5, flag_fragmented=True, flag_fragmented_constant=-1)
- property contours
- feed(labels, border=1, min_area=1, max_area=None, **kwargs)
Notes
May apply inplace changes to
labels
.
- Parameters:
labels – Single label image. E.g. of shape (height, width, channels).
border
min_area
max_area
- property fourier
- property locations
- property reduced_labels
- property resampled_contours
Returns: Tensor[num_contours, num_points, 2]
- property sampled_contours
Returns: Tensor[num_contours, num_points, 2]
- property sampled_sizes
Notes
The quality of sizes depends on how accurate sampled_contours represents the actual contours.
- Returns:
Tensor[num_contours, 2]. Contains height and width for each contour.
- property sampling
- clip_contour_(contour, size)
- contours2boxes(contours)
Contours to boxes.
- Parameters:
contours – Array[num_contours, num_points, 2]. (x, y) format.
- Returns:
Array[num_contours, 4]. (x0, y0, x1, y1) format.
- contours2labels(contours, size, rounded=True, clip=True, initial_depth=1, gap=3, dtype='int32', ioa_thresh=None, sort_by=None, sort_descending=True, return_indices=False)
Contours to labels.
Convert contours to label image.
Notes
~137 ms for contours.shape=(1284, 128, 2), size=(1000, 1000).
Label images come with channels, as contours may assign pixels to multiple objects. Since such multi-assignments cannot be easily encoded in a channel-free label image, channels are used. To remove channels refer to resolve_label_channels.
- Parameters:
contours – Contours of a single image. Array[num_contours, num_points, 2] or List[Array[num_points, 2]].
size – Label image size. (height, width).
rounded – Whether to round contour coordinates.
clip – Whether to clip contour coordinates to given size.
initial_depth – Initial number of channels. More channels are used if necessary.
gap – Gap between instances.
dtype – Data type of label image.
ioa_thresh – Intersection over area threshold. Skip contours that have an intersection over own area (i.e. area of contour that already contains a label vs. area of contour) greater ioa_thresh, compared to the union of all contours painted before. Note that the order of contours is relevant, as contours are processed iteratively. IoA of 0 means no labels present so far, IoA of 1. means the entire contour area is already covered by other contours.
sort_by – Optional Array used to sort contours. Note, that if this option is used, labels and contour indices no longer correspond.
sort_descending – Whether to sort by descending.
return_indices – Whether to return indices.
- Returns:
Array[height, width, channels]. Since contours may assign pixels to multiple objects, the label image comes with channels. To remove channels refer to resolve_label_channels.
- contours2overlay(contours, size, hue_range=(0, 180), saturation_range=(60, 133), value_range=(180, 256), rounded=True, clip=True, intermediate_dtype='float16')
- contours2properties(contours, *properties, round=True, **kwargs)
Contours to properties.
References
[1] https://scikit-image.org/docs/stable/api/skimage.measure.html#skimage.measure.regionprops
- Parameters:
contours – Contours.
*properties – Property names. See [1] for details.
round – Whether to round contours. Default is True.
**kwargs – Keyword arguments for skimage.measure.regionprops.
- Returns:
List of property lists.
- draw_contours(canvas, contours, val=(51, 255, 51), round=True, contour_idx=-1, thickness=2, **kwargs)
- filter_contours_by_intensity(img, contours, min_intensity=None, max_intensity=200, aggregate='mean')
- masks2labels(masks, connectivity=8, label_axis=2, count=False, reduce=<function max>, keepdims=True, **kwargs)
Masks to labels.
Notes
~ 11.7 ms for Array[25, 256, 256]. For same array skimage.measure.label takes ~ 17.9 ms.
- Parameters:
masks – List[Array[height, width]] or Array[num_masks, height, width]
connectivity – 8 or 4 for 8-way or 4-way connectivity respectively
label_axis – Axis used for stacking label maps. One per mask.
count – Whether to count and return the number of components.
reduce – Callable used to reduce label_axis. If set to None, label_axis will not be reduced. Can be used if instances do not overlap.
**kwargs – Kwargs for cv2.connectedComponents.
- Returns:
labels or (labels, count)
- render_contour(contour, val=1, dtype='int32', round=False, reference=None)
- resolve_label_channels(labels, method='dilation', max_iter=999, kernel=(3, 3))
Resolve label channels.
Remove channels from a label image. Pixels that are assigned to exactly one foreground label remain as is. Pixels that are assigned to multiple foreground labels present a conflict, as they cannot be described by a channel-less label image. Such conflicts are resolved by method.
- Parameters:
labels – Label image. Array[h, w, c].
method – Method to resolve overlapping regions.
max_iter – Max iteration.
kernel – Kernel.
- Returns:
Labels with channels removed. Array[h, w].
Eval Operations
- class LabelMatcher(inputs=None, targets=None, iou_thresh=None, zero_division='warn', epsilon=1e-12)
Evaluation of a label image with a target label image.
Simple interface to evaluate a label image with a target label image with different metrics and IOU thresholds.
The IOU threshold is the minimum IOU that two objects must have to be counted as a match. Each target object can be matched with at most one inferred object and vice versa.
Initialize LabelMatcher object.
- Parameters:
inputs – Input labels. Array[height, width, channels].
targets – Target labels. Array[height, width, channels].
iou_thresh – IOU threshold.
zero_division – One of (‘warn’, 0, 1). Sets the default return value for ZeroDivisionErrors. The default ‘warn’ will show a warning and return 0. For example: If there are no true positives and no false positives, precision will return the value of zero_division and optionally show a warning.
- property f1
- property false_negative_labels
- property false_negatives
- property false_positive_labels
- property false_positives
- filter_and_threshold(*a, **k)
- property fowlkes_mallows
- property iou_thresh
- property jaccard
- property precision
- property recall
- property true_positive_labels
- property true_positives
- update(inputs, targets, iou_thresh=None)
- class LabelMatcherList(*args, epsilon=1e-12, rank=None, num_ranks=None, device=None, cache=False, **kwargs)
Label Matcher List.
Simple interface to get averaged results from a list of LabelMatcher objects.
Note
Distributed use assumes, that each example is shown exactly once. Duplicates are not removed. Check your sampler accordingly. Also make sure that each rank calls the same methods in the same order.
Examples
>>> lml = LabelMatcherList([ ... LabelMatcher(pred_labels_0, target_labels0), ... LabelMatcher(pred_labels_1, target_labels1), ... ]) >>> lml.iou_thresh = 0.5 # set iou_thresh for all LabelMatcher objects >>> print('Average F1 score for iou threshold 0.5:', lml.avg_f1) Average F1 score for iou threshold 0.5: 0.92
>>> # Testing different IOU thresholds: >>> for lml.iou_thresh in (.5, .75): ... print('thresh:', lml.iou_thresh, ' f1:', lml.avg_f1) thresh: 0.5 f1: 0.92 thresh: 0.75 f1: 0.91
- Parameters:
*args
epsilon
rank – Rank (e.g. `trainer.global_rank). Allows for distributed communication. If not passed, results will only be computed locally. If passed results are synced across all ranks.
num_ranks – Number of ranks (e.g. trainer.world_size). Allows for distributed communication. If not passed, results will only be computed locally. If passed results are synced across all ranks.
cache – Whether to cache aggregated results. Currently only for distributed environments.
**kwargs
- append(_LabelMatcherList__object)
- property avg_f1
Average F1 score.
- property avg_fowlkes_mallows
- property avg_jaccard
Average Jaccard index.
- property avg_precision
Average precision.
- property avg_recall
Average recall.
- clear()
- clear_cache()
- copy()
- property distributed
- extend(_LabelMatcherList__iterable)
- property f1
F1 score from average recall and precision.
- property f1_np
F1 score from negatives and positives.
- property false_negatives
- property false_positives
- property fowlkes_mallows_np
- insert(_LabelMatcherList__index, _LabelMatcherList__object)
- property iou_thresh
Gets local unique IOU thresholds from all items, if there is only one unique threshold, it is returned.
- property jaccard_np
- property length: int
- pop(*args, **kwargs)
- property precision
- property recall
- property true_positives
- get_pos_labels(v)
- intersection_mask(a, b)
- labels2counts(a)
- labels_exist(func)
- matching_labels(a, b)
- vec2matches(v)
Toy Data
- random_circle(image, mask, x, y, color, radius_range=(3, 28))
- random_ellipse(image, mask, x, y, color, radius_range=(3, 28))
- random_geometric_objects(height=256, width=256, radius_range=(3, 28), intensity_range=(0, 180), margin=13)
- random_rectangle(image, mask, x, y, color, radius_range=(3, 28))
- random_triangle(image, mask, x, y, color, radius_range=(3, 28))