Adding some Hardcoded examples in Generations through BigCode Harness Code
₹600-2000 INR
Cerrado
Publicado hace 11 meses
₹600-2000 INR
Pagado a la entrega
The code generation LM evaluation harness is a framework designed to evaluate code generation models. It is inspired by the EleutherAI/lm-evaluation-harness, which focuses on evaluating language models in general. The evaluation harness supports various code generation tasks and provides features such as multi-GPU text generation, Docker support for security and reproducibility, and compatibility with autoregressive models available on the Hugging Face model hub.
Code generation Python tasks: The harness includes four code generation Python tasks, namely HumanEval, APPS, MBPP, and DS-1000. These tasks can be evaluated using both completion (left-to-right) and insertion (FIM) modes.
The task is to modify the [login to view URL] file in the lm_eval package to append hard-coded prompts and solutions from the HumanEval dataset (below link) to the TokenizedDataset.
[login to view URL]