Depiction of artificial intelligence controlling a human and vice versa.

getty

Anthropic launched two updated versions of its Claude AI large language model on Tuesday. Perhaps more interestingly, it also enabled enhanced functionality that allows users to grant the LLM access and control over certain aspects of their personal computers.

“A vast amount of modern work happens via computers. Enabling Claude to interact directly with computer software in the same way people do will unlock a huge range of applications that simply aren’t possible for the current generation of AI assistants,” wrote a company spokesperson in an email exchange.

Anthropic has dubbed this new computer-navigating capability for Claude “computer use.” While the LLM will not train on any user data per se, the announcement states this is a new approach toward training AI computer navigation with help from the public.

“We’re teaching it general computer skills—allowing it to use a wide range of standard tools and software programs designed for people. Developers can use this nascent capability to automate repetitive processes, build and test software and conduct open-ended tasks like research,” the statement reads.

Claude AI’s Computer Use Capabilities Explained

The company’s announcement included a demo video that showed various functionalities of the software responding to a typed task request from a human. In the demo, once Claude was downloaded to a computer, it was able to do the following:

Take multiple screenshots of a computer screen and analyze the images based on the query request.
It was then able to search for relevant requested information on the demo computer. When it couldn’t locate the requested details on an open file, it accessed other possible databases or sources as instructed in the human-generated request.
Once the information was found, the LLM automatically entered the relevant details into a third-party form/application that was open on the screen.
Lastly, the Claude computer use model went ahead and automatically submitted the completed document.

All steps were accomplished without any intervention by the user once they established the initial request query on their own desktop.

This is important because it’s a significant leap in AI automation access for average individuals, which could further accelerate adoption of AI agents to autonomously complete tasks for us in the future.

Here’s what it looks like in action.

The Anthropic announcement states that this new utility is available for free for interested users who proactively access Claude’s application programming interface for these features and give the LLM access to a personal computer.

Anthropic reiterates several times in its statement that since this software is so new users can expect it to make some mistakes in these early days and be a bit buggy.

“While we expect this capability to improve rapidly in the coming months, Claude’s current ability to use computers is imperfect. Some actions that people perform effortlessly—scrolling, dragging, zooming—currently present challenges for Claude and we encourage developers to begin exploration with low-risk tasks,” the statement reads.

Potential Risks Of Claude AI’s Computer Use

While this AI advancement has created a lot of excitement among AI influencers and developers across social media, there are inherent risks that could arise that Anthropic points out in its post.

“Because computer use may provide a new vector for more familiar threats such as spam, misinformation, or fraud, we’re taking a proactive approach to promote its safe deployment. We’ve developed new classifiers that can identify when computer use is being used and whether harm is occurring. You can read more about the research process behind this new skill, along with further discussion of safety measures, in our post on developing computer use,” the company wrote.

Even though the company’s statement specifically lays out that this upgrade is intended for “developers,” the reality is that by visiting Replit.com and getting a subscription — anyone who can search YouTube and cut-and-paste an API key can access Claude’s computer use function.

All they need to do is follow the bulleted steps outlined under the “Getting started” subhead on the Replit screenshot below — and those steps are:

Add your ANTHROPIC_API_KEY to the Secrets pane
Click Run
Watch the AI work in the Output pane
Send commands to it in the Webview

Replit interface to access Claude computer use.

Replit website 10.23.2024

In the email exchange, an Anthropic representative stressed “a number of safeguards and protections” outlined in a separate blog post regarding Claude’s computer use. An excerpt of the email explicitly called out the following precautions the company has taken.

Developer implementation required: Developers will need to build additional tools to enable Claude to take screenshots and perform specific commands, such as move a cursor or type text, on your computer (or virtual computers) in order to use the computer use capability. Claude cannot take these actions on its own.
Data privacy principles: Consistent with our standard approach to data privacy, by default we don’t train our generative AI models on user-submitted data, including any of the screenshots Claude receives. Anthropic only receives screenshots and associated computer use instructions (user prompts and outputs) from the API customer to Claude. We do not collect any other data from a user’s computer.
Additional safeguards: We’ve developed new classifiers and prompt analysis tools to identify potential misuse of computer use features.
Staged rollout: We’ve deliberately chosen not to release this feature on Claude.ai immediately. Instead, we’re starting with a public beta for developers through our API. This allows us to gather valuable feedback and ensure responsible deployment before considering a broader rollout.

And yet despite those actions, the company also lists the following warnings for potential users on its API implementation page — warnings that are fairly sobering.

Anthropic warning to potential users of Claude computer use.

Anthropic website 10.23.24

The fourth point is particularly concerning where it states that anyone using this function needs to make sure the AI understands that it needs to,

“…Ask a human to confirm decisions that may result in meaningful real-world consequences as well as any tasks requiring affirmative consent, such as accepting cookies, executing financial transactions, or agreeing to terms of service.”

Needless to say, developers and casual users should proceed with caution during this beta phase of Claude computer use.

“We will continue to monitor and refine our safety measures as we gather more data and feedback from the developer beta,” they spokesperson concluded.