Skip to content

Conversation

linhaowei1
Copy link
Contributor

@linhaowei1 linhaowei1 commented Aug 30, 2025

  • This change is worth documenting at https://docs.all-hands.dev/
  • Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

End-user friendly description of the problem this fixes or functionality this introduces.

This PR integrates the AlgoTune benchmark into OH. It transforms all 154 of AlgoTune's Python-based algorithm optimization tasks, preserving the original evaluation logic while adding the necessary infrastructure for autonomous agent testing.

AlgoTune is a comprehensive benchmark featuring over 150 widely used functions from mathematics, physics, and computer science. For each function, the goal is to write Python code that outperforms a reference implementation while producing identical outputs.

The benchmark covers several key domains:

  • Mathematical Algorithms: Matrix operations, numerical methods, linear algebra
  • Signal Processing: FFT/DCT, filtering, convolution
  • Cryptography: Encryption, hashing, security primitives
  • Graph Algorithms: Shortest paths, spanning trees, classic graph problems
  • Optimization: Linear programming, portfolio optimization, scheduling
  • Scientific Computing: ODE/PDE solvers, N-body simulations

Summarize what the PR does, explaining any non-trivial design decisions.

  1. an adapter is implemented to directly transform tasks from original repo to OH repo;
  2. run python run_adapter.py --output-path ../tasks to get all tasks and save them in algotune/tasks;
  3. add readme and script for running OH agent on AlgoTune.

Link of any specific issues this addresses:

Copy link
Contributor

@neubig neubig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing this, it's exciting!

@li-boxuan
Copy link
Collaborator

Could you please fix the linting errors?

Copy link
Collaborator

@li-boxuan li-boxuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't run by myself but looks very clean and well-documented, thank you!

@linhaowei1
Copy link
Contributor Author

Oh sorry, let me fix the linting error locally.

@neubig
Copy link
Contributor

neubig commented Sep 4, 2025

@linhaowei1 could you please check the linting errors and fix (you can install pre-commit and run it to fix most automatically)

@linhaowei1
Copy link
Contributor Author

@linhaowei1 could you please check the linting errors and fix (you can install pre-commit and run it to fix most automatically)

Hi @neubig , can you run the checks again? I think I've done with pre-commit auto fix.

@neubig neubig enabled auto-merge (squash) September 4, 2025 17:28
@neubig neubig merged commit bd8b1bf into All-Hands-AI:main Sep 4, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants