selecting such a varied dataset mix is to develop a language model well suited to perform a wide range
of software engineering tasks, not only those directly related to programming like code completion.
Furthermore, our training data also contains general text datasets in order to provide the model with a
broader linguistic knowledge and context. We hope this enables the model to address a wider range
of queries and tasks in a conversational manner. In Table 1, we provide the data sources, epochs,
categories, and sampling weights of the datasets used to set up the pretraining corpus. We use an
80:20 split distribution of code and natural language data, respectively, with the contributions from
individual components detailed in the table (see Table 2 for references).
Table 2: References for main training datasets.
Dataset Reference
StarCoder Data [26]
Github Issues [25]
Github Diffs [28]
Stackexchange [16]
Arxiv [16]
Synthetic Dataset Sec 2.1.1
Proof Pile Math [6]
Meta Math QA [42]
Refined Web [30]
2.1.1 Synthetic Dataset
We also introduce a small synthetic dataset into our pre-training corpus. The data is synthetically
generated from the seed prompts of the CodeAlpaca
§
dataset, which is comprised of 174,000 prompts.
To augment the diversity and difficulty presented in the code-alpaca prompts, we employed the
"Evol-Instruct" method as introduced by Xu et al., [
40
] wherein we ask a language model (in this case,
we use WizardLM [
40
]) to increase the complexity of the given seed prompt in a step-by-step fashion.
By applying strategies focused on breadth, reasoning, deepening, and complexity, we were able to
enrich our collection with an additional 100,000 prompts. We leverage the DeepSeek Coder 34B
model [
21
] to generate synthetic outputs for the newly developed "Evol-Instruct" prompts. We believe
that introducing this synthetic data early during the pretraining phase helped the model respond better
to natural language text based on an ablation experiment we conducted.
2.2 Long Context Dataset
Building upon the initial pre-training phase, we composed an additional stage of training that
specifically targets the model’s capability to process and comprehend long sequences. Having a
longer context length is useful for coding models due to the usual inter-dependence of multiple
files within a repository. We specifically chose 16,384 as the context length for our long context
dataset after determining the median and mean number of tokens in a software repository to be
≈ 12k
and
≈ 18k
tokens, respectively. This continued training stage focused on a curated selection of
programming languages, all sourced from The Starcoder dataset [
26
], a filtered version of the Stack
which is a large collection of high quality and permissively licensed coding data [
25
]. The languages
selected for this phase were based on the Stack Overflow Developer Survey 2022 [
35
]. In particular,
we selected Python, C, C++, Go, Java, and JavaScript.
To create this long context dataset, we took files written in these languages within a repository
and combined them, inserting a special <repo_continuation> token between each file to maintain
separation while preserving the flow of content. To circumvent any potential biases that might arise
from a fixed ordering of files—a factor that could inadvertently teach the model an unintended
sequence or hierarchy—we employed a randomized strategy. For each repository, we generated not
one, but two distinct orderings of the concatenated files. The statistics are outlined in Table 3.
§
Subset found here - https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K
3
剩余13页未读，继续阅读
猫头虎
- 粉丝: 30w+
- 资源: 462
下载权益
电子书特权
VIP文章
课程特权
VIP享7折，此内容立减5.97元 开通VIP
我的内容管理
展开
最新资源
- 最优条件下三次B样条小波边缘检测算子研究
- 深入解析：wav文件格式结构
- JIRA系统配置指南：代理与SSL设置
- 入门必备：电阻电容识别全解析
- U盘制作启动盘：详细教程解决无光驱装系统难题
- Eclipse快捷键大全：提升开发效率的必备秘籍
- C++ Primer Plus中文版：深入学习C++编程必备
- Eclipse常用快捷键汇总与操作指南
- JavaScript作用域解析与面向对象基础
- 软通动力Java笔试题解析
- 自定义标签配置与使用指南
- Android Intent深度解析：组件通信与广播机制
- 增强MyEclipse代码提示功能设置教程
- x86下VMware环境中Openwrt编译与LuCI集成指南
- S3C2440A嵌入式终端电源管理系统设计探讨
- Intel DTCP-IP技术在数字家庭中的内容保护
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
