21st May, 2011

An incomplete list of some goings on:

• The Sanchay website has been moved to a different server. There were two main reasons for this. The first was that certain things (like online Java applications) can’t be hosted on Sourceforge.net (where the site was earlier hosted) and the online apps were already hosted on another server. The other reason was that Sourceforge.net doesn’t allow outgoing emails, which means that if I create some user’s account on the site (where the user is not a Sourceforge.net user), the confirmation mail won’t be sent to that user.
• In the Syntactic Annotation Interface, it is now possible to build a dependency tree directly on lexical items (words), rather than on chunks. You can, of course, still use the chunk mode, which is the one being used for the major treebank projects for Indian languages.
• There have been some extensions to the corpus query language (Sandhaan). The website for Sandhaan is also being moved, though there not much content there.
• There is now a facility that can give you accumulated statistics for syntactically annotated data. You can query it for specific words, tags, relations etc. For example, if you want to check what tags have been given so far for a selected word, you can do that. You can do the opposite too, i.e., you can what words have been assigned a given tag (say, the tag given to the current word in the Syntactic Annotation Interface). The same for chunks and chunk tags as well as chunks and chunk relations etc.
• There is a version of the validation tool that now uses Sandhaan queries, instead of programs or scripts in Perl or some other language.
• All these changes were made sometime ago, so I don’t remember what other changes there were….

23rd February, 2011

A new avatar (no connection to the movie) of Sanchay is going to appear soon.

The first Sanchay Online Application is on the way. Like most of the major parts of Sanchay, it was done in three-four days.

So that’s why there are so many bugs?

Watch this space (as they say)…

(Also note that the latest builds now will be available here, instead of here.)

19th January, 2011

(Version: 0.4.2)

Syntactic Annotation:

• You can now use different tagsets for different languages. As an example, the list of POS tags for English will have to be stored in the file pos-tags-eng.txt (in the directory Sanchay/workspace/syn-annotation). The default list (used when language specific list for a particular language is not available) will be in the file pos-tags.txt. Similarly for chunk tags (phrase-names-eng.txt). When you select the language in the annotation interface, the tagset for that language (if available) will be used. Otherwise, the default tagset will be used.
• It is also possible now to use hierarchical tagsets. Such tagsets are stored in the same location as the normal tagsets (as above). The levels in tags are specified with double underscore. So, for example, if you want three levels of verb tags (from the most general or coarse to the most specific or fine), you will have to list the tag as something like V1__V2__V3.
• Hierarchical tagsets are used thus: For a given category (say, verb), you can select the level that you are going to use for annotation. For other categories, you can use other levels. The idea is that the annotation can be done at different levels of generality or coarseness for different categories, depending on the purpose for which the annotation corpus is going to be used. For using the second level for verb while annotating a particular corpus, you will select V2 as the desired level before you start annotating. This selection can be done from the syntactic annotation interface by right clicking and selecting Hierarchical Tags. The selection is stored in a mapping file (e.g. pos-tags-levels.txt) in the workspace/syn-annotation directory.
• Duplicate attributes with the same name will not be added now (a bug fix).
• A few other bugs fixed.

XML for Syntactically Annotated Corpus:

• This is major change in the version. Earlier, syntactically annotated corpus (and also some other kinds of corpus) was stored in the SSF format (as the default). Now you can open and save files in XML format too.
• The XML tags used for the XML format are listed in the file xml-props.txt in the Sanchay/props directory.
• XML format will work wherever SSF files were used earlier.

Bugs in Automatic Name Allocation :

• In the annotation interfaces where SSF files were used, the uniques names are automatically generated to identify the nodes. This name generation had some bugs like not handling characters like apostrophe and quotes etc.
• I have tried to fix most of these bugs. If something remains, let me know.

Word Alignment and Sentence Interfaces:

• It is now possible to save in the GIZA++ format (reading this format was available earlier).
• The problems related to wrong handling of apostrophe etc. should not happen now.
• Duplicate alignments (due to duplicate attributes) should also not appear.

Sanchay Corpus Query Language:

• Some bugs fixed in handling queries involving the referred and referring nodes (the R and T nodes).
• Support for the logical operator NOT added. Thus, you should now be able to write something like: C.t=’NN’ and !(C.A.t=’NP’)

10th April, 2010

Some more information about the task mode of operation for the Annotation Interfaces (not yet implemented for the Parallel Corpus Markup):

• The annotation process starts with one copy of the document to be annotated, say, story-1.
• When an annotator (say, john) claims this document (opens and saves it in the task mode), another copy is created, with the file name story-1-john. This is the file on which the annotator will be working.
• However, the task name (e.g. story-1) will apply to all the copies.
• When another user (e.g. terry) has claimed the same task, another copy will be created (story-1-terry).
• Now the adjudicators can use the annotation comparison facility to compare the two annotations.
• The adjudicators can select one of the two annotations and make changes to it directly from the comparison facility.
• When an adjudicator (say, chapman) saves the selected and modified version of one of the annotations, it gets saved with the original document name (story-1), i.e., the original document is overwritten.
• An option worth considering is whether, instead of overwriting the original document, another copy should be created with the name of the adjudicator (story-1-chapman). But what if the adjudicator is also one of the annotators? That shouldn’t be a problem, because it can’t be true for the same document.
• However, the copies of the work by the two annotators remain available for more work by the annotators or for any other use later, such as calculating inter-annotator agreement.

That brings me to the facility I will try to add soon: calculating inter-annotator agreement for different levels of annotation.

Also, note that the task mode of working is not yet available for sentence alignment and word alignment interfaces. That is another item on the agenda.

9th April, 2010

Version 0.4.1 released on Sourceforge.net.

There are the following major changes/additions in this release:

1. Propbank Annotation is a separate application now, instead of being a mode of operation in the Syntactic Annotation Interface. The two letter code is PB.
2. Both the Propbank Annotation and the Syntactic Annotation interfaces can now work in two modes: file mode and the task mode, which can be specified in the the file Sanchay/props/client-modes.txt. You can change it according to your needs.
3. In the task mode, the dormant but previously active facilities of annotation comparison and task generation are available. They are quite simple and should be easy to use.
4. In the task mode, new facilities include the ability to specify two users and two adjudicators for every annotation task. In the beginning, all the tasks (which have been created) are available to everyone, but as soon as two users claim them (by opening and saving them), they become unavailable to others. Only the adjudicators can use the comparison (annotation diff) facility. Look for a file like Sanchay/workspace/syn-annotation/Premchand/Premchand-list.txt, which lists the tasks belonging to the task group named Premchand (one of the greatest Hindi writers). There is a task properties file for every task in the task group, which specifies the details used by the annotation interface. Generating tasks means specifying the files to be annotated and automatically generating the task properties files and the task list file.

If you intend to use the task mode of annotation, which is what I would recommend, then you should try to use the Task Generation facility. You can access it in the task mode, when you select the Setup mode (the other modes being the Work mode and the Compare mode). Once you go through a task list file, such as the one mentioned above, and the task properties files generated in the Setup mode, you will get a fairly good idea about how the task mode operates.

The assumption for the task mode right now is that Sanchay will be located on one computer (preferably Linux: I am not sure whether it can work on Windows for multiple users) and that computer will have accounts for every user who is going to be involved in the annotation process. To make sure that Sanchay is accessible to all these users (write permissions are needed for some properties files), one simple way is to create a group and give read and write permissions to that group for the whole Sanchay directory, except the files you want to have restricted access, e.g. the task list file and client mode file.

2nd April, 2010

Update 1.2. Released only here. Includes the following changes:

1. In the Propbank Annotation Interface (Syntactic Annotation in the Propbank mode), it possible now to navigate by word stem and tags, for which files have to be provided. The default location of these files is Sanchay/workspace/syn-annotation. The file for word stem is named word-navigation-list.txt and the one for tags is named tag-navigation-list.txt. The navigation will work based on the word stem plus tag combination. For example, you can annotate all the main verbs (tag regex ‘”^VM”) with the stem कर (kara: do).
2. In the Propbank mode, some of the dependency structure information will be hidden so that Propbank annotation can be performed without being biased by the dependency structure information and also to ensure that there is less chance of changing this information by mistake. You can still view the dependency tree in the tree visualizer and change it by drag and drop, but that will be fixed in one of the following updates.

In the previous (1.1) update, some further changes were made in the tokenizer so that it shouldn’t split sentence in the case of bullet points (in the case where the bullets are decimal numbers).

29th March, 2010

A couple of bugs fixed in UTF-8 to WX (and vice-versa) conversion. The modified jar file has been uploaded on Sourceforge.net.

28th March, 2010

Update 1 released for the Sanchay version 0.4.0 on Sourceforge.net.

This is a minor update, so you can just download the jar file and replace the old one (in the Sanchay/dist directory) with this one.

There are the following changes/additions in this release:

1. In the Syntactic Annotation interface, when you open a raw text file (simple text without any annotation), the tokenizer that runs will now take into account the old Hindi/Sanskrit sentence/shloka/verse marker (e.g. ।। 1 ।।).

2. You can now convert the encoding of a syntactically annotated file, not just that of a simple text file. This facility has been connected to the Syntactic Annotation interface. (Click on the More button after opening a file).

18th March, 2010

I have released the binary for the 0.4.0 version of Sanchay on Sourceforge. I will release the source code later.

The file released is a zip file containing complete Sanchay. For the next few minor releases, I will only release the jar file (and source code, if possible). The jar file in the current zip file can then be replaced with the new jar file to update your copy of Sanchay from 0.4.0 to, say, 0.4.1. For very minor changes, I will put the jar file on Box.net.

The releases should also be available from the previous location for latest builds.

16th March, 2010

A bug fixed, related to navigation from the result of query processing of multiple files. As I am not able to access the usual site where I put the latest versions, I am putting the jar file with the modification here:

Sanchay jar file only (16th March, 2010)

You should also be able to see the Box.net widget on the side panel on the right.

I am not able to put the complete zip file as the Box.net site account that I have allows files only up to 25 MB.