rannet package¶

Submodules¶

rannet.dataloader module¶

DataLoader for pretraininng

class rannet.dataloader.BertMlmDataLoader(tokenizer: RanNetWordPieceTokenizer, word_segment: Callable, mask_rate: float = 0.15, max_length: int = 512)¶

Bases: DataLoader

DataLoader with BERT MLM setting

process_sentence(obj: str) → Tuple[List[int], List[int]]¶

class rannet.dataloader.DataLoader(tokenizer: RanNetWordPieceTokenizer, max_length: int = 512)¶

Bases: object

dataloader for pretraning

get_random_token(token_id: int) → int¶

static load_tfrecord(record_paths, batch_size, sequence_length=512, buffer_size=None)¶: load dataset from tfrecord

process(corpus: List[List], record_path: str, workers=4)¶: process corpus

process_paragraph(texts: List[str] | List[Dict[str, str]])¶

Parameters:: texts – Union[List[str], List[Dict[str, str]]] for NOLAN-Style: [{“word”: “xxx”, “sentence”: “xxx”}, ], for BERT-Style: [“sentence 1”, “xxx”, ]

process_sentence(text) → Tuple[List[int], List[int]]¶

tfrecord_serialize(instances, instance_keys=['token_ids', 'mask_ids'])¶: convert to tfrecord

truncate_pad_sequence(sequence: List[int], padding_value=0) → List[int]¶

class rannet.dataloader.Seq2SeqLMDataLoader(tokenizer: RanNetWordPieceTokenizer, mask_rate: float = 0.15, max_source_length: int | None = None, max_target_length: int | None = None)¶

Bases: object

DataLoader for seq2seq

process(source_text: str, target_text: str) → Tuple[List[int], List[int], List[int]]¶

rannet.dataloader.subfinder(array: List, sub_array: List) → List[int]¶: find sub-array positions example: >>> array = [0, 0, 1, 2, 3, 5, 1, 2, 3, 1, 2] >>> sub_array = [1, 2, 3] >>> subfinder(array, sub_array) [2, 6]

rannet.layers module¶

class rannet.layers.AdaptiveEmbedding(*args, **kwargs)¶

Bases: Layer

Turns positive integers (indexes) into dense vectors of fixed size. # Arguments

# Input shape

2D tensor with shape: (batch_size, sequence_length).

# Output shape

3D tensor with shape: (batch_size, sequence_length, output_dim).

# References

[Efficient softmax approximation for GPUs](https://arxiv.org/pdf/1609.04309.pdf)

build(input_shape)¶

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:: input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(inputs, **kwargs)¶

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask=None)¶

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

compute_output_shape(input_shape)¶

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

get_config()¶

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:: Python dictionary.

static get_custom_objects()¶

class rannet.layers.AdaptiveSoftmax(*args, **kwargs)¶

Bases: Layer

Turns dense vectors into probabilities. # Arguments

# Input shape

3D tensor with shape: (batch_size, sequence_length, input_dim).

# Output shape

3D tensor with shape: (batch_size, sequence_length, output_dim).

# References

[Efficient softmax approximation for GPUs](https://arxiv.org/pdf/1609.04309.pdf)

build(input_shape)¶

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:: input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(inputs, **kwargs)¶

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask=None)¶

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

compute_output_shape(input_shape)¶

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

get_config()¶

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:: Python dictionary.

static get_custom_objects()¶

class rannet.layers.WithPerplexity(*args, **kwargs)¶

Bases: Layer

call(inputs)¶

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask=None)¶

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

compute_output_shape(input_shape)¶

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

static get_custom_objects()¶

class rannet.layers.WithSparseCategoricalAccuracy(*args, **kwargs)¶

Bases: Layer

call(inputs)¶

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask=None)¶

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

compute_output_shape(input_shape)¶

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

static get_custom_objects()¶

class rannet.layers.WithSparseCategoricalCrossEntropy(*args, **kwargs)¶

Bases: Layer

call(inputs)¶

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

compute_mask(inputs, mask=None)¶

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

compute_output_shape(input_shape)¶

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

static get_custom_objects()¶

rannet.layers.swish(x: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor) → List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor¶

rannet.optimizer module¶

class rannet.optimizer.AdamWarmup(learning_rate: float = 0.001, beta_1: float = 0.9, beta_2: float = 0.999, amsgrad: bool = False, decay: float = 0.0, weight_decay: float = 0.0, epsilon: float = 1e-07, lr_schedule: Dict[int, float] | None = None, gradient_accumulation_steps: int = None, exclude_weight_decay_pattern: List[str] | None = None, include_weight_decay_pattern: List[str] | None = None, **kwargs)¶

Bases: Optimizer

get_config()¶

Returns the config of the optimizer.

An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration.

Subclass optimizer should override this method to include other hyperparameters.

Returns:: Python dictionary.

static get_custom_objects()¶

get_updates(loss, params)¶

class rannet.optimizer.AdamWarmupTF(learning_rate: float = 0.001, beta_1: float = 0.9, beta_2: float = 0.999, epsilon: float = 1e-07, weight_decay: float = 0.0, bias_correction: float = True, lr_schedule: Dict[int, float] | None = None, gradient_accumulation_steps: int = None, exclude_weight_decay_pattern: List[str] | None = None, include_weight_decay_pattern: List[str] | None = None, name: str = 'AdamWarmupTF', **kwargs)¶

Bases: Optimizer

tf keras adam warmup Modified from: https://github.com/bojone/bert4keras/blob/master/bert4keras/optimizers.py#L14

get_config()¶

Returns the config of the optimizer.

An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration.

Subclass optimizer should override this method to include other hyperparameters.

Returns:: Python dictionary.

static get_custom_objects()¶

rannet.optimizer.piecewise_linear(t: int, schedule: Dict[int, float], from_zero: bool = True)¶

piecewise linear modified from:

Parameters:

t – int, iterations
schedule – Dict[int, float], e.g., for {1000: 1, 2000: 0.1}, when t ∈ [0, 1000], ratio increase from 0.0 to 1.0 uniformly, when t ∈ [1000, 2000], ratio decrease from 1.0 to 0.1 evenly, when t > 2000, ratio keep 0.1

rannet.optimizer.symbolic(f)¶

rannet.pretrain module¶

rannet.ran module¶

Implementation of Recurrent Attention Network

class rannet.ran.GatedLinearUnit(*args, **kwargs)¶

Bases: Layer

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:: input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

compute_output_shape(input_shape)¶

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

get_config() → dict¶

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:: Python dictionary.

static get_custom_objects() → dict¶

class rannet.ran.PosMultiHeadAttention(*args, **kwargs)¶

Bases: Layer

build(input_shape)¶

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:: input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(inputs, mask=None, attn_bias=None, **kwargs)¶

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

compute_attention(inputs, mask=None, attn_bias=None)¶

compute_mask(inputs, mask=None)¶

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

compute_output_shape(input_shape)¶

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

get_config()¶

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:: Python dictionary.

static get_custom_objects() → dict¶

class rannet.ran.RAN(*args, **kwargs)¶

Bases: Layer

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:: input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

get_config()¶

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:: Python dictionary.

static get_custom_objects() → Dict¶

class rannet.ran.SelfAttention(*args, **kwargs)¶

Bases: Layer

Creates the variables of the layer (for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:: input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

called automatically before call() executes for the first time.

Parameters:

inputs –
Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

arguments, and inputs cannot be provided via the default value of a keyword argument.
- NumPy array or Python scalar values in inputs get cast as tensors.
- Keras mask metadata is only collected from inputs.
- Layers are built (build(input_shape) method) using shape info from inputs only.
- input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using inputs only.
- Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.
*args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.
**kwargs –
Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

whether the call is meant for training or inference.
- mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

Computes an output mask tensor.

Parameters:

inputs – Tensor or list of tensors.
mask – Tensor or list of tensors.

Returns:

None or a tensor (or list of tensors,: one per output tensor of the layer).

Computes the output shape of the layer.

This method will cause the layer’s state to be built, if that has not happened before. This requires that the layer will later be used with inputs that match the input shape provided here.

Parameters:: input_shape – Shape tuple (tuple of integers) or tf.TensorShape, or structure of shape tuples / tf.TensorShape instances (one per output tensor of the layer). Shape tuples can include None for free dimensions, instead of an integer.
Returns:: A tf.TensorShape instance or structure of tf.TensorShape instances.

get_config() → dict¶

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:: Python dictionary.

static get_custom_objects() → dict¶

rannet.ran.align(tensor: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, axes: int, ndim: int | None = None) → List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor¶

rannet.ran.apply_rotary_position_embeddings(sinusoidal: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, *tensors)¶: apply RoPE modified from: https://github.com/bojone/bert4keras/blob/master/bert4keras/backend.py#L310

rannet.ran.ran(inputs: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, encode_attn: PosMultiHeadAttention, cell: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | None = None, segments: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | None = None, cell_initializer: Callable | None = None, cell_glu: Callable | None = None, cell_residual_layernorm: Callable | None = None, mask: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | None = None, apply_lm_mask: bool = False, apply_seq2seq_mask: bool = False, window_size: int = 128, concat_layernorm=None, memory_review=None, dropout_rate: float = 0.0, min_window_size: int = 16, cell_pooling: str = 'last')¶: Core implementation

rannet.ran.sequence_masking(x: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, mask: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | None = None, value: str | float = '-inf', axis: int | None = None) → List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor¶: mask sequence :param x: input tensor :param mask: mask of input tensor

rannet.rannet module¶

lastest version

class rannet.rannet.RanNet(params: RanNetParams, return_sequences: bool = True, return_cell: bool = True, return_history: bool = False, mlm_softmax: bool = False, apply_cell_transform: bool = True, cell_transform_pooling: str = 'max', apply_lm_mask: bool = False, apply_seq2seq_mask: bool = False, apply_memory_review: bool = True, cell_pooling: str = 'last', min_window_size: int | None = None, window_size: int | None = None, prefix: str = '')¶

Bases: object

check_var_status(key: str, val: Any) → bool¶

static compile(model: Model, learning_rate: float = 0.001, weight_decay: float = 0.01, loss: Dict | Callable | str = 'sparse_categorical_crossentropy', lr_schedule: Dict[int, float] | None = None, gradient_accumulation_steps: int | None = None, **kwargs)¶

encode(x: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, x_mask: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, cell: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | None = None, segments: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | None = None) → List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | List[List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor]¶

static export_checkpoint(config_path: str, ckpt_path: str, target_path: str)¶

Export the default checkpoint to a concise checkpoint removing: redundant variables and assigning meaningful names.

Parameters:

config_path – str, default config path
ckpt_path – str, default checkpoint path
target_path – str, target checkpoint path

static fields_to_check()¶: define fields to be check in export checkpoint

get_inputs(with_cell: bool = False, with_segment: bool = False)¶

get_weight_name(name: str) → str¶

static load_rannet(config_path: str, checkpoint_path: str, window_size: int | None = None, with_mlm: bool = False, with_cell: bool = False, **kwargs) → Tuple[object, Model]¶

Load pretrained RanNet model :param config_path: str. Path to config :param checkpoint_path: str. Path to checkpoint :param with_mlm: bool. Wether to return mlm output. Defaults to False :param kwargs: Other kwargs of RanNet

Returns:: RanNet object model: RanNet keras model
Return type:: RanNet

remove_prefix(name: str) → str¶

restore_weights_from_checkpoint(model: Model, checkpoint_path: str, variable_mapping: Dict | None = None, ran_layers: int = 2)¶: Restore weights from checkpoint :param model: Models. Keras Model :param checkpoint_path: str. Path to checkpoint :param variable_mapping: Optional[Dict]. Variable mapping. Defaults to None, use the default mapping.

static variable_mapping(prefix: str = 'rannet', ran_layers: int = 2) → Dict¶

class rannet.rannet.RanNetForAdaptiveLM(params: RanNetParams, cutoffs: List[int], div_val: int = 1, output_dropout_rate: float = 0.0, return_cell: bool = False, prefix: str = '')¶: Bases: RanNet

class rannet.rannet.RanNetForLM(params: RanNetParams, return_cell: bool = False, prefix: str = '')¶: Bases: RanNet

class rannet.rannet.RanNetForMLMPretrain(params: RanNetParams, **kwargs)¶: Bases: RanNet

class rannet.rannet.RanNetForSeq2Seq(params: RanNetParams, prefix: str = '')¶: Bases: RanNet

class rannet.rannet.RanNetParams(config: Dict)¶

Bases: object

static from_file(config_path: str)¶

rannet.tokenizer module¶

class rannet.tokenizer.RanNetWordPieceTokenizer(vocab: str | Dict[str, int] | None = None, special_tokens: SpecialTokens | None = None, clean_text: bool = True, handle_chinese_chars: bool = True, strip_accents: bool | None = None, lowercase: bool = True, wordpieces_prefix: str = '##')¶

Bases: BaseTokenizer

RanNet WordPiece Tokenizer

static from_file(vocab: str, **kwargs)¶

rematch_to_text(offsets: List[Tuple[int, int]]) → List[List[int]]¶

>>> text = 'hello [PAD] world'
>>> t = tokenizer.encode(text)
>>> mapping = tokenizer.rematch_to_text(t.offsets)
>>> for ch_pos in mapping:
        print(text[ch_pos[0]: ch_pos[-1]+1])
hello
[PAD]
world

train(files: str | List[str], vocab_size: int = 30000, min_frequency: int = 2, limit_alphabet: int = 1000, initial_alphabet: List[str] = [], special_tokens: SpecialTokens | None = None, show_progress: bool = True, wordpieces_prefix: str = '##')¶: Train the model using the given files

train_from_iterator(iterator: Iterator[str] | Iterator[Iterator[str]], vocab_size: int = 30000, min_frequency: int = 2, limit_alphabet: int = 1000, initial_alphabet: List[str] = [], special_tokens: SpecialTokens | None = None, show_progress: bool = True, wordpieces_prefix: str = '##', length: int | None = None)¶: Train the model using the given iterator

class rannet.tokenizer.SpecialTokens(unused_num: int = 1000)¶

Bases: object

property tokens: List[str]¶

rannet.utils module¶

rannet.utils.mean(x: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, mask: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor | None = None, axis: float = -1, keepdims: bool = False) → List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor¶

Generate prefix causal mask :param segment: segment ids

Examples

for segment [[0, 0, 0, 1, 1]], the mask is; array([[[1., 1., 1., 0., 0.],

rannet.utils.standard_normalize(x: List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor, epsilon: float = 1e-07) → List[float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | list] | tuple | float | int | float16 | float32 | float64 | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | ndarray | Tensor | SparseTensor | Variable | KerasTensor¶

Generate triangular causal mask :param seq_len: sequence len

Examples

for seq_len = 3, the mask is: array([[1., 0., 0.],

rannet package¶

Submodules¶

rannet.dataloader module¶

rannet.layers module¶

rannet.optimizer module¶

rannet.pretrain module¶

rannet.ran module¶

rannet.rannet module¶

rannet.tokenizer module¶

rannet.utils module¶

Module contents¶

rannet

Navigation

Related Topics