WISE 2013 Challenge
1. Introduction
WISE 2013 Challenge has two tracks: Entity Linking Track (T1), which is based on the wikilinks dataset, and Weibo Prediction Track (T2), which is based on Sina Weibo dataset. Attendees may attend one or both tracks. A championship and a runner-up will be announced based on submitted result. Selected reports will be published in conference proceedings after review.
Important dates:
* Attendance registration deadline: 24th May, 2013
* Result submission deadline: 7th June, 2013
* Report submission deadline: 7th June, 2013
* Winners notified: 3rd July, 2013
Contact:
For Entity Linking Track (T1):WISE2013ChallengeT1@gmail.com
For Weibo Prediction Track (T2): WISE2013ChallengeT2@gmail.com
2. T1: Entity Linking Track
WISE 2013 Challenge T1 is to label entities within plain texts based on a given set of entities. The set of entities used for labeling are extracted from the Wikilinks dataset. In this challenge, the attendants will be given a test corpus (in English) of plain texts, and a list of entities (distinct Wikipedia URLs extracted from the Wikilinks dataset). Attendees are required to automatically label the mentioned entities or concrete concepts within the given test corpus, using the given Wikipedia URLs. Results will be evaluated based on the precision of entities and concrete concepts that have been correctly labeled.
2.1 Data
The Wikilinks dataset contains around 3M entity names (Wikipedia URLs) and their 40M mentions. Since a certain percentage of Wikipedia URLs in the current Wikilinks dataset are in wrong format, a revised version of the dataset is used. To accomplish the task for entity linking, the attendants need to download the entity list directly from: http://deke.ruc.edu.cn/wisechallenge/entities.zip
The text corpus for testing will be released at the last week of May 2013. Exampling test files are plain texts of the webpages provided by Wikilinks.
The attendants are however allowed to use any other datasets, as long as the linked entities in the submitted results are from the provided dataset.
2.2 Task of T1
Attendees need to find solutions to automatically detect mentions of entities, and link the detected mentions to entities in the given entity file. Detection of proper nouns such as Shanghai, Google Translate, IBM, is relatively easy. For concepts (that typically as classes of entities), attendants are suggested to ignore those common concepts such as trip, book, hotels, because they will be excluded from evaluation whether they are correctly labeled or not. However, concrete concepts such as "Chinese characters", "guidebooks", "online travel agencies" will be considered for evaluation. It is sometimes not that easy to distinguish between common concepts and concrete concepts. The attendants are suggested to label a concept as concrete as possible. For example, the concept ''online travel agencies'' will be better than ''travel agencies''. If a concrete concept has a linking entity, the label to a relatively general concept will be treated as a negative label. Below is an example of selected mentions (denoted by a pair of *) considered for labeling, although some of them may not have linking entities in Wikilinks.
----------------------------------------------------------------
Getting ready for a trip through *China*, especially planning anything outside major cities like *Shanghai* and *Nanjing*, is unlike planning a trip elsewhere. The usual sources--*guidebooks*, *Web searches*, user *review sites*--either don't provide the information you need or are in *Chinese*. Compared with, say, *Southeast Asia*, *China* has not been overrun, and thus well-documented, by independent travelers.
So what should you do? First, learn how to use *Google Translate*. It works and is cheap to use, even if you have to turn on *international roaming*. Second, get a *phrase book* in which key phrases are written out in large-font *Chinese characters*. (Don't count on your *Mandarin pronunciation*.) Third, learn how to count from 1 to 10 on (mostly) one hand as the *Chinese* do: this is how prices will be relayed you. (*Online videos* can help.)
Fourth, familiarize yourself with the English pages of *Elong.com* and *Ctrip.com*, China-based *online travel agencies*, where you can find endless English-language listings for cheap Chinese hotels. I usually paid about $20 a night.
-----------------------------------------------------------------
2.3 Submission guideline
Attendees should submit both results and a report.
1) Results:
Attendees are expected to submit results of a solution in a single file, with each line representing a label of a mention to an entity in the following format:
file_id mention_start_pos entity_URL mention
The file_id is the id (given in the file name) of a test file. The mention_start_pos is the offset of the first character of the entity mention in the test file. The entity_URL is the Wikipedia URL (by removing the prefix ''http://en.wikipedia.org/wiki/'') of the linked entity in Wikilinks dataset. The mention is the detected entity mention in the test file. For example, a test file 100.txt starts with a sentence List of the Oscars 2009. The offset of the mention Oscars 2009 is 12. The correct label for this mention should be:
100 12 81st_Academy_Awards Oscars 2009
Note that mentions of any two labels cannot overlap within a test file. Attendees are recommended to submit their labeling results sorted by file_id and mention_start_pos. An example of a test file and its labeling results can be downloaded from http://deke.ruc.edu.cn/wisechallenge/example.zip.
Attendees are required to submit their final result file(s) to WISE2013ChallengeT1@gmail.com by 7th June, 2013. Each team is allowed to summit at most 4 runs. Attendees are recommended to zip their submission file(s) and name the file as their team name.
2) Report:
Report should be submitted via the WISE 2013 submission system. Attendees should register their submission by 24th May, 2013, and submit the final report by 7th June, 2013. The report should follow the WISE 2013 research paper format requirements. Details of how the attendees finish the challenge tasks should be introduced in the report, while the results should be summarized.
2.4 Evaluation
Although a large number of text files will be provided for testing, only a small percentage of them (more than one thousand however) will be used for evaluation. These selected files are labeled manually by experts, to generate a set of labels as ground truth. Note that both proper nous and concrete concepts are labeled in the ground truth label set. We further generate a subset of ground truth which contains only labels of proper nouns. It is called proper nouns label set.
The submissions will be evaluated based on the percentage of the correct labels covering the full ground truth label set, and the percentage of the correct labels covering the proper nouns label set. Therefore, each run will be evaluated using two scores, one for the labeling precision of both concrete concepts and proper nouns, and the other for the labeling precision of proper nouns only.
3. T2: Weibo Prediction Track
T2 is based on a dataset collected from one of the most popular micro-blog service (http://weibo.com) in China.
3.1 Data
The original data was crawled from Sina Weibo (http://weibo.com), a popular micro-blogging service in China, via the API provided and web crawling. The data distributed in WISE 2013 Challenge is preprocessed as follows:
1) User IDs are anoymized.
2) Tweet contents are preprocessed: retweet content is removed; user name mentioned is removed; only original tweets are preserved and segmented (no meaningful words are removed too).
3) Contents with Chinese words are preprocessed: we map each Chinese word to a unique number. So there are only alphanumeric contents in the dataset.
The dataset that to be used in the track contains three files:
1. User Info: It includes basic information about users (user ID, tags, jobs, personal description, age, gender, education etc.).
2. Tweets: It includes basic information about tweets (user ID, timestamp, tweet content).
3. User Birth: It includes the user's birthday. Notice: This file is only provided in training dataset. You may find some users' birthdays may far away from what is normal. However, your solution should not be interfered.
It should be noted that the dataset is not complete, yet is only a sample of the whole data in the Sina micro-blogging service. In addition, a small dataset with the same setting (without user birth file) is provided for testing purpose. The details of dataset format are given in Appendix 1: Data format
The dataset can be downloaded from:
http://7.yeezhao.com/wise2013challengeT2/
or following links:
1) trainIDBIRTH.zip: http://s.yunio.com/EIJnek
2) trainInfos.zip: http://s.yunio.com/TvcQ_r
3) traintweets.zip: http://s.yunio.com/e4J_7B
4) testinfos.zip: http://s.yunio.com/HxkWxA
5) testtweets.zip: http://s.yunio.com/Wv8lSb
3.2 Task
Attendees are required to build a system for evaluating Weibo users' age range. There are four ranges (1, 2, 3, and 4) in total. Each represents by an integer number. Specifically,
1) Range 1 represents the age interval [0-18],
2) Range 2 represents the age interval (18-24],
3) Range 3 represents the age interval (24-35], and
4) Range 4 represents the ages above 35.
Every user's age falls into one of the four ranges. The goal of this task is to achieve high accuracy.
3.3 Submission guideline
1) Results should be submitted via email to WISE2013ChallengeT2@gmail.com
2) Email tile should have the prefix: WISE2013 Challenge Track 2_
3) All results should be submitted as attachments of the email. Each mail should only contain one attachment whose size should be no more than 1MB.
4) The attachment should be in plain text format with 17519 rows, in which each row contains 2 fields (separated by a tab): the user id and the predicted range value. Use "\n" as the line separator.
3.4 Appendix 1: Data format
There are three files in total. Please open them with Encoding UTF-8, and the line separator is "\n".
1. tweets.txt
Contains a mass of lines. Each line is corresponding to one user and his tweets contents.
A sample line may look like below:
user_id
2. infos.txt
Contains a mass of lines. Each line is corresponding to one user and his personal information.
A sample line may look like below:
user_id
3. idbirth.txt
Contains a mass of lines. Each line is corresponding to one user and his birthday.
A sample line may look like below:
user_id
PS:
We use number 1 or 2 to represents male or female (in gender field).
Documents
Links
trainIDBIRTH.zip: http://s.yunio.com/EIJnek
trainInfos.zip: http://s.yunio.com/TvcQ_r
traintweets.zip: http://s.yunio.com/e4J_7B
testinfos.zip: http://s.yunio.com/HxkWxA
testtweets.zip: http://s.yunio.com/Wv8lSb
