Interacting with pages

This tutorial explains how to interact with pages using Harmony. Harmony is capable of filling out the forms, clicking elements like buttons and checkboxes, and sending keyboard events to the page. Due to browser limitations, it’s not always possible to fully simulate browser events so some events might not be available.

Example: Mailbox cleaner assistant

The use-case is to ask an assistant to select unimportant emails and then mark them as read or remove them, depending on your email workflow. The following prompt would instruct the agent to select the automatic messages:

You are the assistant who helps me read emails and clean my mailbox. Your goal
for this task is to select the automatic messages from the system that require
reading the subject only.

# Step 1

Look at the emails from the current page using the tool `get_google_mail_emails`
and select only those, that are send by the systems (non-humans) automatically
as a notification that does not require further action. Be more aggressive in
deciding what the notification is and assume that most of them do not require
further actions except reading the subject. Make sure to look at each email, do
not ignore any single one.

These notifications may (but not only) include:

* Discounts
* Welcome messages
* New events schedules
* Any marketing communication
* Any sender that looks like a name of the service
* Changes in the terms of service
* Shared documents
* Orders confirmation
* Receipts
* Etc.

# Step 2

Click on the checkbox ids of the selected messages using the tool
`interact_with_page` to select these emails in the UI using `interaction:
click` and `selector: [id="ID"]` (e.g. `selector: [id=":xb"]`). I'm going to
review and remove them manually.

# Step 3

Print a short report of how many notifications were selected and the main
reasons for the selection.