-
Notifications
You must be signed in to change notification settings - Fork 7.6k
Add a new benchmark: AlgoTune #10724
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
use 7 workers to align with tb. 7 is because 154 % 7 == 0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for contributing this, it's exciting!
Could you please fix the linting errors? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't run by myself but looks very clean and well-documented, thank you!
Oh sorry, let me fix the linting error locally. |
@linhaowei1 could you please check the linting errors and fix (you can install pre-commit and run it to fix most automatically) |
Hi @neubig , can you run the checks again? I think I've done with pre-commit auto fix. |
End-user friendly description of the problem this fixes or functionality this introduces.
This PR integrates the AlgoTune benchmark into OH. It transforms all 154 of AlgoTune's Python-based algorithm optimization tasks, preserving the original evaluation logic while adding the necessary infrastructure for autonomous agent testing.
AlgoTune is a comprehensive benchmark featuring over 150 widely used functions from mathematics, physics, and computer science. For each function, the goal is to write Python code that outperforms a reference implementation while producing identical outputs.
The benchmark covers several key domains:
Summarize what the PR does, explaining any non-trivial design decisions.
python run_adapter.py --output-path ../tasks
to get all tasks and save them inalgotune/tasks
;Link of any specific issues this addresses: