class datalinks.api.DLConfig(host, apikey, index, namespace, objectname)
Bases: objectDLConfig class is a configuration container for managing the required settings to interact with DataLinks. It loads configuration values from environment variables to provide flexibility across different environments.This class is designed to simplify the initialization and storage of connection and namespace details required to communicate with DataLinks.
Variables:
host – The host URL for the data layer connection.
apikey – The API key for authentication with the data layer.
index – The index name to be used in the data layer operations.
namespace – The namespace for organizing data in the data layer.
objectname – The name of the object associated with the configuration. Defaults to an empty string.
class datalinks.api.IngestionResult(successful, failed)
Bases: objectRepresents the result of a data ingestion process into DataLinks.This class is a data structure used to store the results of a data ingestion operation. It separates the successfully ingested items from the failed ones, enabling users to track and handle both cases effectively.
Variables:
successful – A list of records successfully ingested. Each record is represented as a dictionary.
failed – A list of records that failed ingestion. Each record is represented as a dictionary.
Bases: objectClass for interfacing with the DataLinks API.Provides methods for ingesting data, managing namespaces, and querying data from DataLinks. Designed to interact with a configurable backend, providing flexibility for deployment environments.
Variables:config – Configuration object containing API key, host, index, namespace, and object name.
Ingests data into the namespace by batching the given data and performing multiple retries in case of failures. This function sends data in chunks (batches), to be processed through configured inference steps, and to resolve entities based on the provided configuration. If a batch fails, it is retried up to a maximum number of attempts.
Parameters:
data (List[Dict[str, Any]]) – List of dictionaries, where each dictionary represents a data block to be ingested.
inference_steps (Pipeline | None) – Pipeline of inference steps to be applied for processing the data. If None the data will be ingested as is.
entity_resolution (MatchTypeConfig | None) – Configuration specifying how entity resolution is to be performed.
batch_size – Number of data blocks to be included in each batch. Defaults to the size of the entire dataset if not provided.
max_attempts – Maximum number of retry attempts for failed batches. Defaults to the provided constant MAX_INGEST_ATTEMPTS.
Creates a new space with the specified privacy settings. This function sends a POST request to create a namespace with the given privacy status. Information about the namespace creation will be logged, including the HTTP status code and response reason. If the namespace already exists, a warning will be logged.
Parameters:is_private (bool) – Determines whether the created namespace will be private or public.
Return type:None
Returns: None
Raises:HTTPError – If the HTTP request fails due to connectivity issues or server-side problems.
Retrieves the list of datasets for the user, optionally filtered by a specific namespace.
Parameters:namespace (Optional *[*AnyStr]) – Optional namespace to filter the datasets by. If provided, only datasets associated with the given namespace will be returned. If not provided, all datasets are retrieved.
Returns: A list of datasets represented as dictionaries if the query is successful and returns a status code of 200, or None if the query fails or encounters an error.
Queries data from a specified data source and processes the response.The method allows querying with a specific query string or with a wildcard (“*”) for all data. The response from the query can be filtered to exclude metadata fields if include_metadata is set to False. Metadata fields are identified by key names starting with an underscore.
Parameters:
query (str) – The query string to use for fetching data. Defaults to “*”, which retrieves all data.
model (str) – The model name to use for inference.
provider (str) – The provider of the LLM model (ollama, openai, etc)
include_metadata (bool) – Specifies whether to include metadata fields in the returned data. Defaults to False.
Returns: A list of records represented as dictionaries, or None if the query fails or an exception occurs during the request.
Return type: List[Dict] | None
Raises:requests.exceptions.RequestException – If a request-related error occurs during querying.
Talk to your data with natural language using DataLinks AutoRAG agent.The method allows asking a question with natural language using our AutoRAG agent. The question will be processed into the necessary steps to answer it.
Parameters:
query (str) – The natural language query to use for fetching data.
model (str) – The model name to use for inference.
provider (str) – The provider of the LLM model (ollama, openai, etc)
include_reasoning (bool) – Whether to include the reasoning steps in the response.
Returns: A tuple with the agent response and the reasoning steps (dict) if include_reasoning is True, or just the response otherwise.
Return type: Any
Raises:requests.exceptions.RequestException – If a request-related error occurs during querying.
class datalinks.cli.StandardCLI(name=‘datalinks-client’, description=‘Infer and link your data!’)
Bases: objectCommand-Line Interface (CLI) wrapper for customizable argument parsing.Simplifies the creation and usage of the DataLinks CLI by allowing to pass a custom callback function for additional arguments specific to an application. It provides a standard set of CLI arguments while enabling customization through user-defined groups.
Bases: StrEnumEnumerates the various resolution strategies for handling matching or reconciliation of entity data. Each enumeration value specifies a particular method or approach used for determining entity equivalence or correspondence.
Variables:
ExactMatch – Used when entities are determined to be equivalent based on exact value matches without any approximation.
GeoMatch – Used when entities are matched based on their geographical location or proximity.
class datalinks.links.MatchTypeConfig(exact_match=None, geo_match=None)
Bases: objectEncapsulates configuration related to different types of entity resolution matches.This class is designed to store, manage, and provide access to various entity resolution match type configurations, such as ExactMatch and GeoMatch. It maintains internal state for these match types and also provides access to a consolidated configuration in dictionary format.
Variables:matchTypes – A dictionary mapping entity resolution types to their respective match configurations (e.g., ExactMatch, GeoMatch).
Bases: ABCAbstract base class for loading resources from a specified folder. It serves as a template for loading files or other resources while maintaining consistency across different implementations.
Variables:folder – Path to the folder from which resources will be loaded.
Bases: LoaderA loader for processing JSON files in a specified folder.Iterates through all .json files within a given folder, parses their content, and processes each JSON object into a standardized format using the load_item method.
Variables:folder – Path to the folder containing JSON files. All .json files in this folder will be processed.
Bases: EnumRepresents different types of processing steps for data manipulation.This class enumerates various distinct processing types that can be used in DataLinks workflows. Each enumeration value signifies a specific stage in the broader data-processing pipeline.
Bases: EnumEnumeration for normalization modes.This class represents different modes of data normalization used in the ‘normalize’ step. It provides three options for normalization: ‘embeddings’ for embedding-level normalization, ‘all-in-one’ for holistic normalization, and ‘field-by-field’ for column-wise normalization.
Variables:
EMBEDDINGS – Mode for normalizing data on an embedding level.
ALL_IN_ONE – Mode for normalizing data holistically, treating the entire dataset as a single entity.
FIELD_BY_FIELD – Mode for normalizing data column-by-column, focusing on individual fields independently.
Bases: EnumEnumeration class that defines various validation modes.This class is designed to specify the modes of operation for the ‘validate’ step. The predefined modes include validation by rows, regular expressions, and fields.
Variables:
ROWS – Validation mode that focuses on rows.
REGEX – Validation mode that utilizes regular expressions.
Bases: objectRepresents the base step within DataLinks.This class serves as the foundational step structure for various implementations. It includes methods to transform its data representation into a dictionary format, custom-processed with specific rules for attributes of Enum type. It is primarily designed as a metaclass.
Variables:step_type – The type of the step, categorized using StepTypes.
class datalinks.pipeline.Normalize(model, provider, target_cols, mode, helper_prompt=”)
Bases: LlmStepUse this step to attempt normalisation of the extracted column names. Table inference across different unstructured data blocks may result in different field names for the same information, hence the need to normalize the column names.Encapsulates the configuration necessary to perform the ‘normalize’ step. It specifies the desired target columns, the mode of normalisation, and includes optional helper prompts to provide further instructions or context.
Variables:
target_cols – A mapping of the desired column names to an optional description used as context.
mode – Specifies the normalisation mode to be applied.
helper_prompt – Optional helper text or prompt information.
