Failure to build tiktoken wheel on python 3.13 using uv

[lily-de]

Problem

While setting up the uv environment for Goose, we encountered an issue with building the tiktoken package on Python 3.13. The error indicates a failure to build the wheel due to the absence of a Rust compiler.

Steps to Reproduce:

1. Create a new virtual environment via uv venv
2. Activate the virtual environment with source .venv/bin/activate
3. Sync dependencies using uv sync

Expected Behavior:

All dependencies, including tiktoken, should be resolved and built correctly.

Actual Behavior:

The build process fails for tiktoken with the following errors:

error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: tiktoken==0.7.0
  Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1

Environment

Python Version: 3.13
Operating System: macOS 14.0 arm64

Additional info

The issue may be related to the recent release of Python 3.13, as tiktoken does not have a precompiled wheel available for this version. Existing virtual environments seem unaffected as they do not require re-building the wheel.

Potential Solutions:

• Downgrade to Python 3.12 until tiktoken is updated.
• Manually install a Rust compiler and attempt to build the wheel from source.
• Update the tiktoken dependency to a version that supports Python 3.13.

Discussion:

• It was suggested to pin Python to 3.12 for now using uv python pin 3.12 or update the tiktoken version for forward compatibility.
• This issue might soon affect other developers as they update or recreate their environments.

Action Items:

• Confirm if pinning Python version or updating tiktoken is more feasible.
• Consider updating the release documentation to warn about potential issues with Python updates.

[lily-de]

Can confirm that uv python pin 3.12 works for fixing this issue locally

[lamchau]

@lily-de i think we can bump the version? https://github.com/openai/tiktoken/releases/tag/0.8.0

[lamchau]

keeping this open so we can bump the version