AI Agents

Solving Complex Tasks with Large Language Models and Tools

Sean Davis, MD, PhD

Thursday, October 17, 2024

Beyond Text: LLMs as Agents

  • Traditional LLMs excel at text generation, but their knowledge is limited to their training data.
  • Tool use allows LLMs to become active agents, interacting with the real world.
  • Examples: retrieving up-to-date information, performing calculations, running code, controlling software.

When RAG is not enough

The Rationale for Tool Use

  • Overcoming Knowledge Limitations: Accessing external data sources like the internet or specific databases keeps the LLM’s knowledge current and relevant.
  • Enhancing Capabilities: Tools allow LLMs to perform tasks they cannot do independently, like booking a flight or controlling a smart home device.
  • Improving Accuracy: Using tools for tasks like calculations or fact-checking ensures more reliable outputs.

Components of LLM Tools

  • API: The interface that allows the LLM to communicate with the tool.
  • Description: A clear definition of the tool’s function, inputs, and outputs. This allows the LLM to understand how to use the tool correctly.
  • Invoker: A mechanism that triggers the tool based on the LLM’s instructions.

What is an API?

  • API stands for Application Programming Interface.
  • It’s a set of rules and specifications that allow one application to access the features or data of another application.
  • This enables different applications to communicate and interact with each other.

APIs and LLM Tools

  • APIs are the foundation of LLM tool use.
  • They provide the communication channel between the LLM and the external tool.
  • The LLM uses the API to send requests to the tool and receive responses.

How LLMs Use Tools

  1. Task Identification: The LLM analyzes the user’s request and determines if a tool is needed.
  2. Tool Selection: The LLM selects the appropriate tool based on the task and the available tools.
  3. API Call Formulation: The LLM generates the correct API call to the selected tool, including any necessary parameters.
  4. Response Integration: The LLM receives the tool’s output and integrates it into its response to the user.

GPTs/Tools

https://openai.com/index/introducing-gpts/

Structured Output (JSON)

  • JSON stands for JavaScript Object Notation.
  • It’s a lightweight data format that is easy for humans to read and write and easy for machines to parse.
  • JSON is commonly used for transmitting data between web applications and servers.

JSON and LLM Tools

  • Many LLM tools use JSON for input and output.
  • The LLM sends requests to the tool in JSON format.
  • The tool responds with results in JSON format, which the LLM can then process.

Benefits of Using JSON

  • Standardized Format: Ensures consistency and compatibility between different tools.
  • Easy to Parse: Allows the LLM to quickly extract the information it needs from the tool’s response.
  • Flexible: Can represent a wide range of data structures.

Agents and complex tasks

LLMs as components of larger systems

Planning

Memory

Tools

Future of LLM Tool Use

  • More Sophisticated Tools: LLMs will be able to use a wider range of tools, including those that can perform more complex tasks.
  • Improved Integration: LLMs will be able to seamlessly integrate tool outputs into their responses.
  • Increased Autonomy: LLMs will be able to make more decisions about when and how to use tools.

LLM Tool Use in Healthcare

  • Potential Applications: Accessing patient records, retrieving medical research, generating reports, scheduling appointments.
  • Benefits: Improved efficiency, accuracy, and decision-making.
  • Challenges: Ensuring patient privacy and data security.