v5.1.0: rate limits, refactored settings

Highlights

In-housed rate limits management

• Centers on a moving window algorithm with either a Redis or in-memory state
• Supports dynamically defined rates for different models or providers.
• New bundled configurations for different OpenAI rate limit tiers
• Accomplished using new third party dependencies coredis and limits

Refactored Settings to allow for increased flexibility

• Indexing
  • Indexes can use relative paths, enabling sharing across machines
  • Paper search now no longer rebuilds the index every invocation
  • Index parameter now are grouped in IndexSettings
    • This release begins a deprecation cycle for the original hyperparameters
  • Index builds now have a rich.Progress bar
• Parsing
  • Chunking and embedding can now be deferred to inference time
• Agents
  • Agents now have a max_timesteps parameter to upper-bound trajectory length
  • Default agent is now a simple tool calling agent (ToolSelector), instead of a deterministic sequence of tool calls ("fake" agent)

Several bug fixes centered on retry-able errors:

• Flaky Semantic Scholar and Crossref SSL errors and connection reset errors
• LLM completions and text embeddings

What's Changed

• Cleaning up #489's implementation by @jamesbraza in #503
• chore(deps): lock file maintenance by @renovate in #504
• chore(deps): lock file maintenance by @renovate in #506
• chore(deps): update all non-major dependencies by @renovate in #505
• Filtering two more DeprecationWarnings by @jamesbraza in #509
• Refactor to create settings.agent.index grouping by @jamesbraza in #510
• Removed extra save_index calls, and added missing changed by @jamesbraza in #513
• Not rebuilding SearchIndex every paper_search by @jamesbraza in #512
• Updated citation to arxiv preprint by @whitead in #514
• Aviary agent max_timesteps and fixed test_gather_evidence_rejects_empty_docs by @jamesbraza in #515
• Moved reset_log_levels to usefixtures by @jamesbraza in #517
• Decomposing Answer.could_not_answer by @jamesbraza in #516
• Fixing IndexSettings.use_absolute_paper_directory leading to relative index file paths by @jamesbraza in #518
• Moving run_ldp_agent to center on RolloutManager by @jamesbraza in #519
• Retrying on known Semantic Scholar flaky SSL error in get_s2_doc_details_from_doi by @jamesbraza in #522
• Converted PyMuPDF message to warning logs by @jamesbraza in #523
• rich.Progress bar for monitoring index builds by @jamesbraza in #521
• Better descriptions and log messages by @jamesbraza in #524
• Made it possible to skip chunking by @whitead in #526
• Retrying on aiohttp.ClientConnectionResetError by @jamesbraza in #529
• Add rate limits for LLMs and Embedding Models by @mskarlin in #520
• Disallowing confusing None from IndexSettings.index_directory, and IndexSettings.get_named_index_directory by @jamesbraza in #531
• Add router_kwargs in separate control flow step by @mskarlin in #532
• Propagating AgentSettings.agent_type default for synchrony by @jamesbraza in #533
• Adding retrying of aembedding if it fails by @jamesbraza in #535
• Add limits+coredis to mypy by @mskarlin in #537
• Lock file maintenance by @renovate in #534
• Controlling for pymupdf version in test_pdf_reader_match_doc_details VCR by @jamesbraza in #538
• Lock file maintenance by @renovate in #539
• Fixed yet another api.semanticscholar.org:443 ssl:default error via retrying by @jamesbraza in #540

Full Changelog: v5.0.10...v5.1.0