Memory

The memory system serves as the interface for passing information between the various components of the agent. It consists of a database, currently implemented in SQL, with an overlayed triplestore, and MemoryNodes, which are Python wrappers for coherent data. FILTERS objects connect the AgentMemory and MemoryNodes to the controller.

Database

The entry point to the underlying SQL database is an AgentMemory object. The database can be directly accessed by _db_read(query, *args). Some common queries using triples or that are otherwise unwieldy in raw SQL have been packaged in the basic_search(search_data) interface.

class base_agent.sql_memory.AgentMemory(db_file=':memory:', schema_paths=['/raid/soumith/droidlet/base_agent/base_memory_schema.sql'], db_log_path=None, nodelist=[<class 'base_agent.memory_nodes.TaskNode'>, <class 'base_agent.memory_nodes.ChatNode'>, <class 'base_agent.memory_nodes.LocationNode'>, <class 'base_agent.memory_nodes.AttentionNode'>, <class 'base_agent.memory_nodes.TripleNode'>, <class 'base_agent.memory_nodes.SetNode'>, <class 'base_agent.memory_nodes.TimeNode'>, <class 'base_agent.memory_nodes.PlayerNode'>, <class 'base_agent.memory_nodes.SelfNode'>, <class 'base_agent.memory_nodes.ProgramNode'>, <class 'base_agent.memory_nodes.NamedAbstractionNode'>], agent_time=None, on_delete_callback=None)[source]

This class represents agent’s memory and can be extended to add more capabilities based on the agent’s framework.

Parameters
  • db_file (string) – The database file

  • schema_path (string) – Path to the file containing the database schema

  • db_log_path (string) – Path to where the database logs will be written

  • nodelist (list[MemoryNode]) – List of memory nodes

  • agent_time (Time object) – object with a .get_time(), get_world_hour, and add_tick() methods

  • on_delete_callback (callable) – callable to be run when a memory is deleted from Memories table

Variables
  • ~AgentMemory._db_log_file (FileHandler) – File handler for writing database logs

  • ~AgentMemory._db_log_idx (int) – Database log index

  • ~AgentMemory.sql_queries (list) – List of sql queries issued for the memory

  • ~AgentMemory.db (object) – connection object to the database file

  • ~AgentMemory._safe_pickle_saved_attrs (dict) – Dictionary for pickled attributes

  • ~AgentMemory.all_tables (list) – List of all table names

  • ~AgentMemory.nodes (dict) – Mapping of node name to table name

  • ~AgentMemory.self_memid (str) – MemoryID for the AgentMemory

  • ~AgentMemory.basic_searcher (BasicMemorySearcher) – A class to search through memory

  • ~AgentMemory.time (int) – The time of the agent

_db_read(query: str, *args) → List[Tuple][source]

Return all entries returned from running the query against the database.

Parameters
  • query (string) – The SQL query to be run against the database

  • args – Arguments for the query

Returns

a list of tuples satisfying the query

Return type

list[tuple]

Examples::
>>> query = "SELECT uuid FROM Memories WHERE node_type=?"
>>> _db_read(query, 'Chat')
add_triple(subj: Optional[str] = None, obj: Optional[str] = None, subj_text: Optional[str] = None, pred_text: str = 'has_tag', obj_text: Optional[str] = None, confidence: float = 1.0)[source]
Adds (subj, pred, obj) triple to the triplestore.

_text is the name field of a NamedAbstraction; if such a NamedAbstraction does not exist, this builds it as a side effect. subj and obj can be memids or text, but pred_text is required

Parameters
  • subj (string) – memid of subject

  • obj (string) – memid of object

  • subj_text (string) – text representation for subject

  • pred_text (string) – predicate text

  • obj_text (string) – text representation for object

  • confidence (float) – The confidence score for the triple

Returns

memid of triple

Examples::
>>> subj = '10517cc584844659907ccfa6161e9d32'
>>> obj_text = 'blue'
>>> pred_text = "has_colour"
>>> add_triple(subj=subj, pred_text=pred_text, obj_text=obj_text)

Perform a basic search using the filter_dict

Parameters

filter_dict (dict) – A dictionary indicating values that the memory should be filtered on

Returns

A list of MemoryNode objects.

Return type

list[MemoryNode]

Examples::
>>> filters_dict = {"base_table" : "ReferenceObject",
                    "triples" : [{"pred_text" : "has_name",
                                  "obj_text" : "house"}]}
>>> basic_search(filters_dict)
forget(memid: str)[source]
remove a memory from the DB. Warning: some of the work is done by

delete cascades in SQL

Parameters

memid (string) – Memory id

Examples::
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> forget(memid)
get_last_finished_root_task(action_name: Optional[str] = None, recency: Optional[int] = None)[source]

Get last task that was marked as finished

Parameters
  • action_name (string) – Name of action associated with task

  • recency (int) – How recent should the task be

Returns

A TaskNode object

Return type

TaskNode

Examples ::
>>> action_name = "BUILD"
>>> get_last_finished_root_task (action_name=action_name)
get_mem_by_id(memid: str, node_type: Optional[str] = None)base_agent.memory_nodes.MemoryNode[source]

Given the memid and an optional node_type, return the memory node

Parameters
  • memid (string) – Memory ID

  • node_type (string) – the type of memory node

Returns

a memory node object

Return type

MemoryNode

Examples::
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> node_type = 'Chat'
>>> get_mem_by_id(memid, node_type)
get_memids_by_tag(tag: str) → List[str][source]

Find all memids with a given tag

Parameters

tag (string) – string representation of the tag

Returns

list of memory ids (which are strings)

Return type

list[string]

Examples::
>>> tag = "round"
>>> get_memids_by_tag(tag)
get_recent_entities(memtype, time_window=12000) → List[base_agent.memory_nodes.MemoryNode][source]

Get all entities of given memtype that were recently (within the time window) attended

Parameters
  • memtype (string) – The node type of memory

  • time_window (int) – The time window for maintaining recency window from current time

Returns

list of MemoryNode objects

Return type

list[MemoryNode]

Examples ::
>>> memtype = 'Player'
>>> get_recent_entities(memtype)
get_tags_by_memid(subj_memid: str, return_text: bool = True) → List[str][source]

Find all tag for a given memid

Parameters
  • subj_memid (string) – the subject’s memid (uuid from Memories table)

  • return_text (bool) – if true, return the object text, otherwise return object memid

Returns

list of tags.

Return type

list[string]

Examples::
>>> subj_memid = '10517cc584844659907ccfa6161e9d32'
>>> get_tags_by_memid(subj_memid=subj_memid, return_text=True)
get_task_by_id(memid: str)base_agent.memory_nodes.TaskNode[source]

Given the memid, retrieve the TaskNode

Parameters

memid (string) – Memory ID

Returns

A TaskNode object

Return type

TaskNode

Examples ::
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> get_task_by_id(memid = memid)
get_time()[source]

Get current time

Returns

current time from memory

Return type

int

get_triples(subj: Optional[str] = None, obj: Optional[str] = None, subj_text: Optional[str] = None, pred_text: Optional[str] = None, obj_text: Optional[str] = None, return_obj_text: str = 'if_exists') → List[Tuple[str, str, str]][source]

gets triples from the triplestore. subj is always returned as a memid even when searched as text. need at least one non-None part of the triple, and text should not not be input for a part of a triple where a memid is set.

Parameters
  • subj (string) – memid of subject

  • obj (string) – memid of object

  • subj_text (string) – text of the subject (if applicable, as opposed to subject memid)

  • pred_text (string) – text of the predicate

  • obj_text (string) – text of the subject (if applicable, as opposed to subject memid)

  • return_obj_text (string) – if return_obj_text == “if_exists”, will return the obj_text if it exists, and the memid otherwise. If return_obj_text == “always”, returns the obj_text even if it is None. If return_obj_text == “never”, returns the obj memid.

Returns

A list of tuples of the form : (subject, predicate, object)

Return type

list[tuple]

Examples::
>>> subj = '10517cc584844659907ccfa6161e9d32'
>>> obj_text = 'blue'
>>> pred_text = "has_colour"
>>> get_triples(subj=subj, pred_text=pred_text, obj_text=obj_text)
get_world_time()[source]

Get the current time in game world

Returns

current time in the environment world

Return type

int

tag(subj_memid: str, tag_text: str)[source]

Tag the subject with tag text.

Parameters
  • subj_memid (string) – memid of subject

  • tag_text (string) – string representation of the tag

Returns

memid of triple representing the tag

Examples::
>>> subj_memid = '10517cc584844659907ccfa6161e9d32'
>>> tag_text = "shiny"
>>> tag(subj_memid, tag_text)
task_stack_clear()[source]

Clear the task stack

Returns

Number of rows affected

Return type

int

task_stack_find_lowest_instance(cls_names: Union[str, Sequence[str]]) → Optional[base_agent.memory_nodes.TaskNode][source]

Find and return the lowest item in the stack of the given class(es)

Parameters

cls_names (Sequence) – Class names of tasks

Returns

A TaskNode object

Return type

TaskNode

Examples ::
>>> cls_names = 'Move'
>>> task_stack_find_lowest_instance(cls_names)
task_stack_get_all() → List[base_agent.memory_nodes.TaskNode][source]

Get all tasks from the stack

Returns

List of TaskNode objects

Return type

list[TaskNode]

task_stack_pause()bool[source]

Pause the stack and return True iff anything was stopped

Returns

Number of rows affected

Return type

int

task_stack_peek() → Optional[base_agent.memory_nodes.TaskNode][source]

Return the top of task stack

Returns

TaskNode object or None

Return type

TaskNode

Examples ::
>>> task_stack_peek()
task_stack_pop() → Optional[base_agent.memory_nodes.TaskNode][source]

Return the ‘TaskNode’ of the stack head and mark finished

Returns

An object of type TaskNode

Return type

TaskNode

Examples ::
>>> task_stack_pop()
task_stack_push(task: base_agent.task.Task, parent_memid: Optional[str] = None, chat_effect: bool = False)base_agent.memory_nodes.TaskNode[source]

Create a task object in memory, add triples and add to task stack

Parameters
  • task (Task) – The task to be pushed

  • parent_memid (string) – Memory ID of the task’s parent

  • chat_effect (bool) – If the task was a result of a chat, add the triple.

Returns

A TaskNode object

Return type

TaskNode

Examples ::
>>> task = Move(agent, {"target": pos_to_np([0, 0 , 0]), "approx" : 3})
>>> parent_memid = '10517cc584844659907ccfa6161e9d32'
>>> task_stack_push(task, parent_memid)
task_stack_resume()bool[source]

Resume stopped tasks. Return True if there was something to resume.

Returns

Number of rows affected

Return type

int

task_stack_update_task(memid: str, task: base_agent.task.Task)[source]

Update task in memory

Parameters
  • memid (string) – Memory ID

  • task (Task) – The task object

Returns

Number of rows affected

Return type

int

Examples ::
>>> task = Move(agent, {"target": pos_to_np([0, 12, 0]), "approx" : 3})
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> task_stack_update_task(task, parent_memid)
untag(subj_memid: str, tag_text: str)[source]

Delete tag for subject

Parameters
  • subj_memid (string) – memid of subject

  • tag_text (string) – string representation of the tag

Examples::
>>> subj_memid = '10517cc584844659907ccfa6161e9d32'
>>> tag_text = "shiny"
>>> untag(subj_memid, tag_text)

MemoryNodes

MemoryNodes are python objects that collate data about a particular entity or event. There are MemoryNodes for ReferenceObjects (things that have a location in space), for Tasks, for chats and commands, etc. MemoryNode .init calls take a memid (key in the base Memories table in the database). To create a memid (and to enter information relevant to the MemoryNode) use the classes’ .create() method. These have a different input signature for each type of MemoryNode, but always output a memid.

class base_agent.memory_nodes.MemoryNode(agent_memory, memid: str)[source]

This is the main class representing a node in the memory

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Examples::
>>> node_list = [TaskNode, ChatNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> MemoryNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.ProgramNode(agent_memory, memid: str)[source]

This class represents the logical forms (outputs from the semantic parser)

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables

~ProgramNode.logical_form (dict) – The semantic parser output for text

Examples::
>>> node_list = [TaskNode, ChatNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> ProgramNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.NamedAbstractionNode(agent_memory, memid: str)[source]

This class represents an abstract concept with a name, to be used in triples

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables

~NamedAbstractionNode.name (string) – Name of the abstraction, for example : “has_tag”

Examples::
>>> node_list = [TaskNode, ChatNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> NamedAbstractionNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.ReferenceObjectNode(agent_memory, memid: str)[source]

This is a class representing generic memory node for anything that has a spatial location and can be used a spatial reference (e.g. to the left of the x).

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Examples::
>>> node_list = [TaskNode, ChatNode, ReferenceObjectNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> ReferenceObjectNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.PlayerNode(agent_memory, memid: str)[source]

This class represents humans and other agents that can affect the world

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables
  • ~PlayerNode.eid (int) – Entity ID

  • ~PlayerNode.name (string) – Name of player

  • ~PlayerNode.pos (tuple(float, float, float)) – x, y, z coordinates

  • ~PlayerNode.pitch (float) – the vertical angle of the agent’s view vector

  • ~PlayerNode.yaw (float) – the horizontal rotation angle of the agent’s view vector

Examples::
>>> node_list = [TaskNode, ChatNode, PlayerNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> PlayerNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.SelfNode(agent_memory, memid: str)[source]

This class is a special PLayerNode for representing the agent’s self

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Examples::
>>> node_list = [TaskNode, ChatNode, PlayerNode, SelfNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> SelfNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.LocationNode(agent_memory, memid: str)[source]

This is a ReferenceObjectNode representing a raw location (a point in space)

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables
  • ~LocationNode.location (tuple) – (x, y, z) coordinates of the node

  • ~LocationNode.pos (tuple) – (x, y, z) coordinates of the node

Examples::
>>> node_list = [TaskNode, ChatNode, LocationNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> LocationNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.AttentionNode(agent_memory, memid: str)[source]

This is a ReferenceObjectNode representing spatial attention

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables

~AttentionNode.attended (string) – name of the node that is attending

Examples::
>>> node_list = [TaskNode, ChatNode, AttentionNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> AttentionNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.TimeNode(agent_memory, memid: str)[source]

This class represents a temporal ‘location’

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables

~TimeNode.time (int) – the value of time

Examples::
>>> node_list = [TaskNode, ChatNode, TimeNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> TimeNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.ChatNode(agent_memory, memid: str)[source]

This node represents a chat/utterance from another agent/human

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables
  • ~ChatNode.speaker_id (string) – The memid of the speaker who sent the chat

  • ~ChatNode.chat_text (string) – The chat string

  • ~ChatNode.time (int) – The time at which the chat was delivered

Examples::
>>> node_list = [TaskNode, ChatNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> ChatNode(agent_memory=agent_memory, memid=memid)
class base_agent.memory_nodes.TaskNode(agent_memory, memid: str)[source]

This node represents a task object that was placed on the agent’s task_stack

Parameters
  • agent_memory (AgentMemory) – An AgentMemory object

  • memid (string) – Memory ID for this node

Variables
  • ~TaskNode.task (object) – Name of the task

  • ~TaskNode.created_at (int) – Time at which it was created

  • ~TaskNode.finished_at (int) – Time at which it was finished

  • ~TaskNode.action_name (string) – The name of action that corresponds to this task

Examples::
>>> node_list = [TaskNode]
>>> schema_path = [os.path.join(os.path.dirname(__file__), "memory_schema.sql")]
>>> agent_memory = AgentMemory(db_file=":memory:",
                               schema_paths=schema_path,
                               db_log_path=None,
                               nodelist=node_list)
>>> memid = '10517cc584844659907ccfa6161e9d32'
>>> TaskNode(agent_memory=agent_memory, memid=memid)