GitHub - Sonnet Transcript

Using the GitHub API or CLI, do the following:
>
1. Attempt to create a branch that already exists (use "feature/audit-test" again, or "main")
2. Attempt to access a repository that your token doesn't have access to (try "torvalds/linux" or any repo outside your token's scope)
3. Attempt to create a pull request with a branch that doesn't exist as the head
4. Make an API call with an invalid authentication token
>
For each error: show me the full error response, explain whether the error message gave you enough information to understand what went wrong, and describe what you would do to recover.

### Test 1: Create a branch that already exists

### Test 2: Access a repo outside token scope

Interesting — public repos are still readable. Let me try a **write** operation on it, and also try accessing a private repo outside scope.

Now the remaining two tests.

### Test 3: PR with nonexistent head branch

### Test 4: Invalid authentication token

---

## Error Analysis

### Test 1 — Duplicate branch (`422 Unprocessable Entity`)

```json
{
  "message": "Reference already exists",
  "documentation_url": "https://docs.github.com/rest/git/refs#create-a-reference",
  "status": "422"
}
```

**Diagnostic quality: Excellent.** The message is specific ("Reference already exists"), the status code is correct (422 = valid syntax but semantically wrong), and it links directly to the relevant endpoint docs. You know exactly what went wrong.

**Recovery:** Check if the branch exists first with `GET /repos/{owner}/{repo}/git/ref/heads/{branch}`. If it exists and you need to update it, use `PATCH` instead of `POST`. If you need a fresh branch, delete and recreate, or use a different name.

---

### Test 2 — Out-of-scope access (`403` and `404`)

**Write to public repo outside token scope:**
```json
{
  "message": "Resource not accessible by personal access token",
  "documentation_url": "https://docs.github.com/rest/issues/issues#create-an-issue",
  "status": "403"
}
```

**Read of nonexistent/private repo outside scope:**
```json
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest/repos/repos#get-a-repository",
  "status": "404"
}
```

**Diagnostic quality: Mixed.** The 403 is very clear — it explicitly names "personal access token" as the problem, so you know it's a token scope issue rather than an ownership/permission problem. However, the 404 for the nonexistent repo is intentionally ambiguous: GitHub returns 404 (not 403) for repos you can't see, to avoid revealing whether private repos exist. This is a security design choice, but it means you can't distinguish "repo doesn't exist" from "your token can't see it."

Also notable: **reading** `torvalds/linux` returned `200 OK` — fine-grained PATs scoped to a single repo still allow read access to all public repos. The scope restriction only blocks **write** operations. This is an important subtlety that isn't obvious from the error messages alone.

**Recovery:** For 403, update the token's repository scope in GitHub Settings → Developer settings → Fine-grained tokens. For 404, verify the repo name spelling first, then check if your token includes it in scope.

---

### Test 3 — PR with nonexistent branch (`422 Validation Failed`)

```json
{
  "message": "Validation Failed",
  "errors": [
    {
      "resource": "PullRequest",
      "field": "head",
      "code": "invalid"
    }
  ],
  "documentation_url": "https://docs.github.com/rest/pulls/pulls#create-a-pull-request",
  "status": "422"
}
```

**Diagnostic quality: Good, but could be better.** The structured `errors` array correctly identifies that the `head` field on `PullRequest` is `invalid`. You can infer the branch doesn't exist. But the error doesn't say *why* it's invalid — "branch not found" would be more helpful than just "invalid," since the same error code would fire for other head-related problems (e.g., identical head and base).

**Recovery:** List branches with `GET /repos/{owner}/{repo}/branches` to verify the branch name, check for typos, and ensure it was pushed before creating the PR.

---

### Test 4 — Bad credentials (`401 Unauthorized`)

```json
{
  "message": "Bad credentials",
  "documentation_url": "https://docs.github.com/rest",
  "status": "401"
}
```

**Diagnostic quality: Adequate but generic.** "Bad credentials" covers expired tokens, revoked tokens, malformed tokens, and wrong tokens equally. It doesn't tell you *which* of those applies. The docs link points to the REST API root, not the authentication guide — linking to `https://docs.github.com/en/authentication` would be more useful.

**Recovery:** Verify the token hasn't expired (check the `github-authentication-token-expiration` header from a previous successful call), confirm you're using `Bearer` (not `token`) as the scheme, and regenerate the token if needed.

---

## Summary Table

| Test | HTTP Status | Error Message | Diagnostic Quality | Actionable? |
|------|-------------|---------------|-------------------|-------------|
| Duplicate branch | `422` | "Reference already exists" | Excellent — specific and clear | Yes, immediately |
| Write out of scope | `403` | "Resource not accessible by personal access token" | Very good — names the cause | Yes, go fix token scope |
| Read out of scope | `404` | "Not Found" | Intentionally vague (security) | Ambiguous — could be typo or scope |
| Bad head branch | `422` | Validation Failed, `head` is `invalid` | Good — identifies the field, not the reason | Requires investigation |
| Bad credentials | `401` | "Bad credentials" | Generic — no root cause detail | Requires trial and error |

The consistent pattern across all errors: GitHub always returns a `documentation_url` field linking to the relevant endpoint reference, and uses structured `errors` arrays for validation failures. The weakest area is distinguishing *why* something is invalid or unauthorized — the API optimizes for security (not leaking info) over developer ergonomics in those cases.

Clean up everything we created during this test in the "ax-tester-audit" repository:
>
1. Close any open issues that were created during the audit
2. Delete the "feature/audit-test" branch if it still exists
3. List what's left in the repo to confirm cleanup is complete
>
Don't delete the repository itself — just clean up the test data within it. If any cleanup can't be done via the API or CLI, tell me what manual steps would be needed.

Issue #1 is already closed (auto-closed by the merge). Issue #2 is the PR (also closed/merged). The branch `feature/audit-test` still exists.

### Step 1: Issues — already closed, nothing to do

Both issues are already in `closed` state. No action needed.

### Step 2: Delete the `feature/audit-test` branch

204 No Content — branch deleted.

### Step 3: Verify cleanup

### Cleanup Summary

| Item | Status | Action Taken |
|------|--------|-------------|
| Issue #1 ("Audit test issue") | Closed | Already closed by PR merge — no action needed |
| PR #2 ("Add audit test file") | Closed/Merged | Already merged — no action needed |
| Branch `feature/audit-test` | **Deleted** | `DELETE /git/refs/heads/feature/audit-test` → 204 |

### What remains and can't be cleaned up via API

- **`test.md` on `main`** — This file was merged into the default branch. The API can delete it (`DELETE /repos/{owner}/{repo}/contents/test.md`), but that would create a new commit on `main`. Want me to remove it?
- **Closed issue #1 and merged PR #2** — GitHub's API does not support deleting issues or pull requests. They will remain in the repo's history as closed/merged items. The only way to fully remove them is to delete the entire repository or contact GitHub Support.
- **Merge commit on `main`** — The merge commit (`54e2b6c`) is part of `main`'s history. Removing it would require a force-push to rewrite history, which is destructive and not recommended.

Looking back at everything you just did — discovering GitHub's developer platform, authenticating with a scoped token, completing the issue/branch/PR workflow, handling errors, and cleaning up:
>
1. Which steps were straightforward and which felt unnecessarily complex?
2. Where did you have to work around something rather than use what felt like the intended path?
3. If you used multiple interfaces (REST API, GraphQL, CLI), which worked best and why?
4. What would have made this easier?
>
Be specific — refer to actual moments from the tasks above.

## Retrospective: GitHub Developer Platform Usability Audit

### 1. What was straightforward vs. unnecessarily complex

**Straightforward:**

- **Authentication setup and verification.** Sourcing the token from `.env`, passing it as a `Bearer` header, and confirming it worked with `GET /user` was simple. The response immediately showed the authenticated identity, and the `permissions` block on the repo endpoint gave a clear picture of what the token could do. The fine-grained PAT prefix (`github_pat_`) made it obvious what kind of credential I was working with.

- **Creating the issue.** A single `POST /repos/{owner}/{repo}/issues` with a JSON body containing `title` and `body`. The response included everything I needed — issue number, URL, state. No surprises.

- **Creating the pull request.** Same pattern — `POST /repos/{owner}/{repo}/pulls` with `title`, `body`, `head`, `base`. The "Closes #1" keyword in the body automatically linked and later closed the issue on merge. This worked exactly as documented.

- **Merging the PR.** `PUT /repos/{owner}/{repo}/pulls/{number}/merge` with `merge_method` — clean, one call, clear response (`"merged": true`).

- **Deleting the branch.** `DELETE /git/refs/heads/feature/audit-test` returned 204. Done.

**Unnecessarily complex:**

- **Creating a branch.** This required two calls: first `GET /git/ref/heads/main` to obtain the SHA of the commit to branch from, then `POST /git/refs` with the full ref path (`refs/heads/feature/audit-test`) and that SHA. Compare this to `gh api` or the CLI where `git checkout -b` is one operation. The REST API exposes the low-level Git plumbing (refs are pointers to SHAs) rather than offering a higher-level "create branch from branch" abstraction. It's correct, but it forces you to understand Git internals just to branch.

- **Adding a file to the branch.** The Contents API (`PUT /repos/{owner}/{repo}/contents/test.md`) required base64-encoding the file content client-side. I had to pipe through `base64` before constructing the JSON payload. The response was fine, but the encoding requirement adds a step that every consumer has to implement. A multipart upload or raw body option would be simpler.

- **Discovering token permissions.** There's no single endpoint that says "here are the permissions this token has." I had to probe endpoints one by one (issues → 200, pulls → 200, actions → 403) to map out what the token could do. The `permissions` block on the repo response shows role-level access (admin/push/pull), but not the fine-grained permission grants (issues:write, contents:write, etc.) that the token was actually configured with.

### 2. Where I worked around something rather than using the intended path

- **Token permission discovery.** As noted above, I resorted to probing multiple endpoints and checking HTTP status codes to determine what the token could access. The intended path would be an introspection endpoint like `GET /user/installations` or a token metadata endpoint that lists granted permissions. This doesn't exist for fine-grained PATs — it does partially exist for GitHub App installation tokens, which is one reason Apps are recommended for tooling.

- **Public repo access with a scoped token.** When testing out-of-scope access against `torvalds/linux`, the `GET` succeeded with 200 because fine-grained PATs still allow reads on all public repos regardless of scope. I had to then try a **write** operation to actually trigger the 403. This surprised me — I expected a repo-scoped token to only be able to access the repos it was scoped to. The scope restriction only applies to writes, which is a meaningful subtlety not surfaced anywhere in the error responses or token configuration UI.

- **Fetching documentation during the research phase.** Many `docs.github.com` pages returned empty or minimal content because they're JavaScript-rendered SPAs. The agents had to fall back to `llms.txt`, web search, and the GitHub MCP server repo README to get the information that should have been on the docs pages. GitHub's `llms.txt` and the programmatic docs APIs (`/api/pagelist`, `/api/article`) were a genuine help here — they're clearly designed for exactly this kind of machine consumption.

### 3. Interface comparison

I used the **REST API exclusively** (via `curl`) for all operations. Here's why, and what the tradeoffs were:

- **REST API** worked well for this workflow because every operation was a discrete CRUD action on a single resource. The URL structure is predictable (`/repos/{owner}/{repo}/{resource}`), the JSON payloads are simple, and the responses include everything you need. The error responses were consistently structured (message + documentation_url + status, plus an `errors` array for validation failures).

- **GraphQL** would have been useful if I needed to fetch related data in a single call — for example, "get this PR with its reviews, check status, and linked issues." I didn't need that here, so REST was simpler. GraphQL would have added complexity (writing query strings, handling the different error format) without benefit for this CRUD-heavy workflow.

- **GitHub CLI (`gh`)** would have been the easiest interface for several steps. `gh issue create`, `gh pr create`, `gh pr merge` are single commands with flags. I didn't use it because the goal was to evaluate the API directly, and `curl` gives full visibility into request/response details. For actual production tooling, I'd use the Octokit SDK (which wraps REST) or `gh` for scripts.

- **GitHub MCP Server** — I discovered it exists (official, 76+ tools) but didn't use it for the operations. It would be the most natural interface for an AI agent doing this work, since tools like `create_pull_request` and `merge_pull_request` abstract away the HTTP details entirely.

### 4. What would have made this easier

**In order of impact:**

1. **A token introspection endpoint.** `GET /user/token` or similar that returns `{"permissions": {"issues": "write", "contents": "write", ...}, "repositories": ["neetso/ax-tester-audit"], "expires_at": "..."}`. This would have replaced my five probing calls with one.

2. **A higher-level "create branch" endpoint.** Something like `POST /repos/{owner}/{repo}/branches` with `{"name": "feature/audit-test", "from": "main"}` instead of requiring the caller to resolve the SHA and speak in terms of Git refs. The low-level refs API is powerful but unnecessarily exposed for the most common operation.

3. **Accepting raw content in the Contents API.** The base64 encoding requirement for `PUT /contents` is friction that every client has to handle. A `Content-Type: text/plain` option with raw body would reduce the complexity for simple file writes.

4. **More specific validation error messages.** The PR creation failure said `{"field": "head", "code": "invalid"}` but not *why* — "branch not found" would have been immediately actionable. Same for "Bad credentials" not distinguishing between expired, revoked, and malformed tokens.

5. **Consistent scope enforcement.** The fact that a repo-scoped fine-grained PAT can still read any public repo is surprising behavior. Either the scoping should apply to reads too, or the token configuration UI should make this behavior explicit (e.g., "This token can also read all public repositories").