Skip to content

utils

convert_examples_to_features

Convert text examples to BERT specific input format. Tokenize the input text and convert into features.

Args
  • examples: Text data.

  • tokenizer: Tokenizer to process the text into tokens.

  • max_seq_length: The maximum length of the text sequence supported.

Returns
  • all_input_ids: ndarray containing the ids for each token.

  • all_input_masks: ndarray containing 1's or 0's based on if the tokens are real or padded.

  • all_segment_ids: ndarray containing all 0's since it is a classification task.

InputFeatures

A single set of features of data.