Working History

2010-04-01 Do Implementing the last changes suggested by ON.

2010-03-29 Mo Last rehearsal of final presentation with AA in the morning. Held the final presentation during the interview. Got some good questions, I think I got my point across. ON gave me feedback to my BA, mostly minor changes... Will fix those asap.


**2010-03-26 Fr** Fixed things at the final presentation, implemented changes suggested by AA. Brought ON a print version of by BA.

**2010-03-24 We** Final proofreading remarks of J. and E. implemented. Last details corrected due to remarks from AA. Committed the stuff to SVN, sent ON a mail that it is ready.

**2010-03-23 Tu** Improved final presentation. Rehearsal with AA, got some tips to improve it. Looks quite fancy atm.

**2010-03-22 Mo** Created first version of the final presentation of my bachelor thesis.

2010-03-19 Fr Went over the whole thing again, proofreading the last chapters.

2010-03-18 Th Proofreading of the whole work, fixing bugs, typos and stuff. Inluded the corrected version of the bender architecture diagram.

2010-03-15 Mo Finishing new toy example section in the trust chapter, looked over the whole thesis again. Looks goot atm, but I need to proofread it still.


**2010-03-12 Th** More writing, tumdidum. New section for trust chapter: A toy example of trust data with a demonstrative calculation of the karma and trust values.

**2010-03-11 Th** More writing, tumdidum.

**2010-03-09 Tu** More writing, tumdidum.

**2010-03-08 Mo** Put the nice graphs I made over the weekend into the thesis. Sweet :). Fixed tables and organisation in the appendices. Reworked introduction and related work chapter. Cleared most todos. Almost doubled number of references for nice bibliography. Printed final draft and gave to to AA.

2010-03-05 Fr Writing, writing, more writing... Prepared data so I can create some nice graphs over the weekend.

2010-03-04 Th Looked at GnuPlot. Bloody extensive, maybe a bit overkill for my 3 graphs... Will try something with Maple first. Wrote more evaluation. Added macrothingy to the tex for Ruby source code and used it in evaluation chapter (RBender part).

2010-03-02 Tu Finished camera ready version of the SRC paper and submitted it. Filled in application form for CAPS funding and sent it of, together with application per mail with all attached files (resume, budget plan, ...). Sent application email to CHOOSE, again with resume, motivation letter, more stuff. Furthermore did some writing on evaluation chapter, implementation of AA's comments, and the RBender chapter. Also did a good new todo-list.

2010-03-01 Mo [Civil Service course preparation for the whole day]


**2010-02-26 Fr** Lot of writing: Evaluation chapter, Appendix B (Metadata from Bender), lot of corrections in JBender chapter... Attended Alumni-Lunch: Good thing!

**2010-02-25 Th** After being sick for 2 days finally back at work... Got feedback from Adrian and will work on implementing it. We also finished the camera-ready version of the Trustability paper according to the reviewers feedback and submitted it.

**2010-02-22 Mo** Little bit of writing, meeting with AA. Inserted comments to the rewiever (@AA) into the thesis...
[Civil Service cadre report in the Afternoon]

2010-02-18 Th Lot of writing. Wrote chapter on RBender. Reorganised section about JBenders architecture. Wrote Subsection about Code Search Result Suitability.

2010-02-15 Mo Implemented previous/next page button on Bender front end to display more results. Also fixed a bug that did not (in all cases) parse the projects name correctly from the path of the found Java source file. Furthermore did some writing as always...


**2010-02-13 Sa** Some thinking about the discrete ranking for Benders trustability metric. Wiki is not as helpful here as expected... Will have to play around with the data a bit to find something that fits our purpose.

**2010-02-09 Tu** More writing on the general chapter about code search. About 3.5 pages on suitability, code bases and human trustability factors.

**2010-02-08 Mo** Implemented sorting by trustability on the JBender frontend, works fine now. (Allthough it does not 'remember' what you last sorted by, which is a bit annoying...) Writing on the general chapter about code search engines. did 2.5 pages in about 2 hours, not that bad... :)

2010-02-05 Fr Lot of writing. Acknowledgments, JBender chapter, Trustability chapter, appendix for JBender, ... Some minor changes in the frontend. And some first research for a ranked trustability function.

2010-02-02 Tu Polished up Bender front end. Integrated the new index (over 200k files with 3.5M terms - woot!), it works like a charm now. Started to collect some top-terms statistics that will make a nice chapter for my thesis' appendix. Fought with automated LaTex table creation - it is a total bitch.

2010-02-01 Mo Bender front end is more or less finished now. Thesis is set up, discussed lineup with Adrian; it should be fine for now... Started rewriting the bender indexer to be scalable enough to handle the 200k Java files we have. Those files are now on my laptop, I will create the index locally.


**2010-01-27 Fr** Worked on the Bender front end. Wrote first things for my thesis and finally fixed the layout issus in LaTex. Started copying all Java files from bender to laptop since local index creation is maybe not faster but a lot more practical... Copying will take some time (19GB data in lots of small files.)

**2010-01-26 Tu** Did some things that came out short during the last days, like writing this working history blag... Organized further work, started with my thesis by just importing the paper into new stylesheet according to university thesis standards...

**2010-01-25 Mo** Was at the dentist in the morning. Numb head an no meeting (ON not around) convinced me of staying at home.

2010-01-22 Fr Was working at ID during the morning. In the afternoon I implemented last changes due to ON's feedback. Then I sent a last version to Adrian. At 02:00 hour I finally found some time for the very very last changes (AA feedback) and was eventually able to submit my paper. Yay for me!

2010-01-21 Th Lot of writing, last number collected to go into the paper. We finished everything and at 23:59 hour I finally had the final draft for a final review by ON. (Send ON a mail; paper is on SVN.)

2010-01-20 We More writing, more playing around with the trust metric function. Did some analysis, we made sure our inputs follow power law distributions. Added all this info to latest paper draft.

2010-01-19 Tu Implemented first version of trustability metric with Adrian. Furthermore I did a lot of writing for the paper, new info, changing and reorganizing things with respect to ON's feedback. (Abstract has be rewritten!)

2010-01-18 Mo We again grabbed all Ohloh files, pimped up speed a bit. (The old files had some errors in them...) Now we have Ohloh statistics for about 3700 projects with 57'000 developers and about 190'000 users. Discussed my idea of a trustability function with AA. We gave our first paper draft to ON for a review...


**2010-01-15 Fr** Small changes in the bender indexing machine. Tried to let it run on bender so we get a big (!) index. Along the way I somehow screwed up my Java VM (or whatever) - nothing seems to compile or run anymore. Fuck! But at least the Ohloh crawler has allready collected enhanced metadate for about 2900 projects.

**2010-01-14 Th** Ohloh crawler generation 2 hacking, while AA pimps up the parser. Will use ActiveRecord to save ohloh meta data. *Deleted jdbc/sqlite3 GEM*
Wrote a small script that downloads social/user metadata for ohloprojects. Ran fine locally, I'm gonna run it on Bender over the weekend. It uses AA's parser and writes out .json files. Hope I will get activerecord DB stuff towork on friday.

**2010-01-10 Mo** More looking into Ohloh, more making plans with AA. Looked over my Ohloh things again, there is a new deadline for a paper> January 19th. So what we are trying to do is the following: Get social/user metadata from ohloh. Create bigger java index. Write automated trustability function. Write two more pages of paper (total of 4 is needed). All in 9 days. Pfeeewww. :/

2010-01-08 Fr Looked at Ohloh again, decided what new data we will need. > More social and user metadata: Users, developers, contributors, their kudos, ratings, ... Looked at the metadata and source code we have yet with AA.

2010-01-07 Th Made battle plans with AA. SCG is not gonna pay for my trip to south africa, but I will send them my paper anyway. Now it's written I can at least get some third pary feedback for it. Furthermore I decided upon following the trustability approach for my further thesis work, and letting the RBender thing aside. (Maybe I can finish that in the summer?) Gave the SRC paper the last twinges and inserted the diagram. AND I FINALLY SUBMITTED IT. WOOT!

2020-01-05 Tu Feeling sick today but showed up anyway because of the deadline. Short meeting with AA in the morning, then went home to sleep.

2010-01-04 Mo More SRC paper writing, more references. Paper is basically finished now. Also made first versions for a (sort of UML) diagram.


**2009-12-29 Tu** Chose some first references for the SRC paper and embedded them in the tex using bibtex. Created Citeulike account and startet collecting papers for my personal bib file.

**2009-12-28 Mo** Wrote first draft for 'Background and Related Work' paragraph for the SRC paper. Looked at papers and references.

~~~<br>

2009-12-08 Tu Working on SRC paper.

2009-12-07 Mo A bit more Java programming at the Ruby end of the Bender Lucene machine. Diving into LaTex stuff, setting up the docs for thesis and the two papers (SRC and MSR).


**2009-12-04 Fr ** Starting to integrate parsed Ruby results into the existing Lucene Bender framework, without comments at the moment. Decided upon what and how to index Ruby entities. Uploaded first thesis structure to svn/Students repo.

**2009-12-03 Th** Further LaTex doodling. Working on the JRuby AST and the not yet available comments.

**2009-11-30 Mo** First glance at JRuby AST work from Rapperswil. Started with a first thesis version, using the layout of Dominique Matter. Wrote the first thesis lineup.

2009-11-26 Th Tried to add comments to the parsed Ruby model but run into a wall there. The comments are not yet properly implemented in the AST of JRuby-parser and in the proper JRuby parser / AST they are not even mentioned. At least I found some work of a Professor in Rapperswil that concerned Ruby comments in the JRuby AST. Hopefully I will be able to profit of his results.

2009-11-22 Mo CHOOSE Forum. Was interesting and fun to be there.


**2009-11-20 Fr** More Ruby Parser hacking. Added code ranges to the parsed modules/classes/methods. Works really quite up to now...

**2009-11-19 Th** Severe Java hacking with AA, we started implementing a Ruby model to represent the Ruby structure we can parse from an AST.

**2009-11-17 Tu** Reorientation with AA, we decided upon using JRuby-Parser for parsing Ruby as a a dynamic programming language. Bender is going dynamic.

2009-11-13 Fr Figured out some Ruby parsers, have still to decide upon which I'm gonna use.

2009-11-08 Mo Looked into further course of my project. We decided upon going along the way of creating a CSE for dynamic languages, specifically Ruby. I am now trying to figure out an easy way to parse Ruby code, preferably from within my existing bender (Java) code.


**2009-11-06 Fr** Decided to use mongrel, looks small and seems easy to use. Got a mongrel server running locally soon enough and made him display the now even better bender page. On bender.unibe.ch there seems to be a problem with the old JRuby (1.1), so I instlled a new one (1.4). Got new JRuby and mongrel to work on bender. Sadly the -deamonize option for mongrel is not compatible with JRuby, had to hack around to get the process running in background. Look here: http://bender.unibe.ch:9090
Port 9090 should open soon hopefully.

**2009-11-05 Th** Got a more funky GUI and some decent outputs for results found for a query. Also had to adapt the Java source so that a search on bender returns enough information in the form of ScoreDocument objects. Started looking into (J)Ruby on Rails server and how I could make them work on bender the server. Bender is now on Twitter as @bender_cse.

**2009-11-02 Mo** Got rails to work with sqlite via jdbc-adapter (install the gem, then put correct adapter in database.yml). Also found a simple to access my Bender Lucene stuff from the frontend. First working prototype is functional as off today! Subscribed with Frabrizio for the CHOOSE Forum.

2009-10-30 FR Rather unproductive day. Tried to set up JRuby, but it wont work with my rails project yet because of some missing sqlite3 gems that are not JRuby compatible. Tried to access my lucene code from JRuby but got stuck there as well. It seems that there are about twenty different ways to use Java from within JRuby and none did work for me. And the official bloody JRuby wiki is down.

2009-10-29 Th Worked on the rails webfrontend. Got a controller that makes a dummy search and displays a (potentially dynamic) list. Tried to switch from Ruby to JRuby, but the rails stuff seems to need some native extention gems (sqlite3...). Will have to look at that again. Once that is done it should be easy to include bender (lucene, Java) and actually do a proper search over the GUI.

2009-10-27 Tu Showed my progress to AA, we discussed the further work. If I manage to get a working web frontend up this week I'm good in my schedule.

2009-10-26 Mo Read some tutorials and tried to set up some Rails stuff... Does not work yet though. But at least I got some base structure for the web frontend app (because it was created automatically) and I managed to get some ruby code to be executed on a click on the 'search' button. Haven't yet found out how I can actually display any results yet, this automagic Rails stuff is rather obscure.


**2009-10-23 Fr** Looking into possibilities to create a smooth GUI for benders search. (Ruby on Rails maybe?) My plan: I want the full prototype up and working (with a suitable GUI) before I go over it again and start out improving things. (Do it. Then do it right. Then do it fast.) Before GUI creation I should also look into the multi field search and other mighty Lucene search options.
...
JRuby on Rails could be a sweet choice for my web frontend. It has the nice rails design and with JRuby I should easily be able to acces the Java Lucene engine. I started on a very first GUI, alltough I don't have anything yet besides a first html page without any functionality.

**2009-10-22 Th** Generated a new local test index using the new parser. The compiler of the parser displays a lot of compilation problems while parsing, but I am not sure as to if and how these problems concern my index. Wrote a first implementation of a search that allows a user to enter a query and presents him with some results. Works like a charm up to know - I'll have to test multi-field and logical searches though.

**2009-10-20 Tu** Further improvement of the new parser, adapted the existing indexing to use it. Worked over the to-be-parsed data and fixed some issues with null values.

**2009-10-19 Mo** Fought through the jungle that is the com.sun.tools.javac compiler package. Not much of an API available out there, but with some help from Lea's code (see notes) I got a nice parser to work. It parses .java files for the basic source code data:<br>
Complete Src, Complete Javadoc, Classnames, Methodnames, Methodbodies (both also sorted by visibility), Implementations, Extensions, Imports, ...

2009-10-16 Fr Yeah, had a breakthrough with bender. A first functional index was created locally, containing around 270 .java files. One can easily browse the index with Luke, a Lucene index browser in java with a nice GUI. (see Notes...)

2009-10-15 Th Heavy coding at bender. Wrote classes to search for java files in the repositories, to parse those files into lucene index documents (very simple parser) and to generate an index over these docs. Was able to create a first index at 1930 in the evening, it does not yet contain any words though... Strange, will have to look at that tomorrow.

2009-10-13 Tu Read more about (free) search engines. Decided to use Lucene, an open source framework/library in Java for highly performant search engines. —> Seems to be fairly potent, and it really takes a lot of my shoulders when it comes to implementing things. At the same time I am still able to do everything myself if I want to, especially all the parsing.
Startet to build a framework for my search engine, it is named 'bender'. Renamed my Ruby things (= metadata crawler) to 'bendy'.

2009-10-12 Mo Looked at the source code download that was running on Bender over the weekend: At the moment about 43k .java files are lying in repos under /home/fgysin/sources/web/ohloh.net/try1. We have some source code repos now, but all together it is really annoyingly slow. In 2 days about 50 project repos were downloaded, with the big ones still waiting out there. Roughly estimated it would take us not less than 4 years to grab the source code for all Java projects from Ohloh...


**2009-10-09 Fr** Added better exception handling and output to the SourceGrabber. SVN, CVS and GIT are supported up to now and should work, provided that the repositories are available and open to public access. After a local testrun I decided to just start the Grabber on Bender and let it run through the weekend... PROBLEM: Download of repos seems to be very unrealiable!

**2009-10-08 Th** Wrote a Ruby script to read the most important metadata from Json files and write it to a sqlite database. This allows proper sorting and searching through the projects. Of the existing 4369 projects only 3785 could be written to the db. Anyway the others most probably have an empty 'enlistments' tag in their Json file and are therefore useless to my cause.

**2009-10-05 Mo** Held my first bachelor thesis presentation. Went quite well, got some questions and a bit of feedback. (Wrote that down.) Am now trying to figure out a way to decide upon which projects I want to fill my core code base with. AA is now gone for 3 Weeks. Looked into the subject of mirroring mercurial repos with GIT - looks like there isn't an easy way to do that, at least not as easy as with CVS/SVN.

2009-10-02 Fr More hacking around with Adrian. Finished OhlohParser, we now have nice Json files with lots of metadata for each OhlohProject.

2009-10-01 Th Some hacking around with Adrian. Worked on sorting and cleaning up the downloads. Furthermore worked on OhlohParser to create Json files for all downloaded OhlohProjects.

2009-09-28 Tu Found out that Ohloh messed up their search so that quite a lot of projects tend to appear in the result list multiple times. The first 10k projects turn out to be only around 4.4k - still a nice number.

2009-09-27 Mo Looked into my first presentation with Adrian. Tried to do some changes on Laptop with OO, but the OO pendant for Powerpoint sucks totally. Not usable at all.


**2009-09-26 Su** Did a small presentation with about 12 slides for my first speech about my bachelor thesis.

**2009-09-24 Fr** Further work on Ohloh downloader. Downloaded the first 10'000 projects from Ohloh. (or rather the meta data for the first 10k...)

**2009-09-22 Tu** Figured out a way for axel to use other proxy config than the env variables: /etc/axelrc. Now axel also works from within ruby calls. Implemented multithreading/threadpooling for the download of Ohloh project data: Severely boosts speed (like 20x) but is not that reliable up to now - it seems that certain threads go missing... [Edit: This probably was a mistake, Ruby multithreading seems to be rather difficult.]

**2009-09-21 Mo** Finally migrated from the old tower to my new laptop - everything is going swiftly now. (And got my IP &mdash;> proper internet)
Started writing a tool to download source code from remote repositories. The goal would be to be able to handle as many repository types as possible: SVN, CVS, GIT, Mercurial, ...
Also added some content to this wiki page to be a little bit more informative.

2009-09-20 Su Set up my laptop with all the needed ruby-gear 'n stuff. Haven't yet recieved a DHCP registration for its mac address though —> can only use the slow public net WLAN.

2009-09-18 Fr Wrote Ohloh.net downloader to collect a list of project names from Ohloh and download the important files. Also got my new laptop and will hopefully be able to work some more during the weekends now...

2009-09-15 Tu Worked on the Ohloh.net parser, it now collects about all the important information from Ohloh for a given project and persists it to Json.

2009-09-14 Mo Tried out JEdit and Aptana-Standalone for Ruby, finally decided to use Aptana Plugin for Eclipse 3.4.2 as my 3.5 Eclipses (including the Aptana-Standalone) fail to recognise the correct proxy configuration. Ordered and paid my new laptop - Lenovo T400 - hope it arrives soon...


**2009-09-08 Tu** Further Project Eulering. PP with AA, small implementation of a parser that parses the Ohloh Enlistments page for project repos. Parsing via Nokogiri, persistence via Json.

**2009-09-07 Mo**
NetBeans is unusable, as is the Groovy plugin for Eclipse. Need another Editor... Tried out JEdit - but the official plugin there is a mess to. I see a better alternative: Groovy is dead - long live Ruby! (First steps in Ruby with the Project Euler...) Maven repo is still mirroring to Bender. Done some clones of SVN and CVS mirrors to bender/local - works great with GIT, but will take an awful lot of time for the larger repos...

2009-09-03 Th GIT mirroring seems to have problems... Also download of the Maven repo is terribly slow - will take a while.

2009-09-01 Tu More looking into NebBeans as a new Groovy Editor, looks nice at first glance. Worked with Git, tried to mirror some SVN repos to GIT repos on Bender. Did not work up to now...

2009-08-31 Mo Maybe we could salvage Metadata from Maven? Started to download complete Maven repository to Bender.


**2009-08-28 Fr**
Got Grape to work (more or less) and first hibernation.

**2009-08-25 Tu**
Finishing new setup, starting examples for Groovy hibernate and Groovy Grape.

**2009-08-24 Mo**
After Laptop crash lot of work went up in smoke. Luckily I was able to save most of my data. Set up new System on tower pc, and started to again with installing and setting up Eclipse/Groovy...

2009-08-21 Fr Crawler can now crawl Ohloh-Project-Enlistenment-Pages and extract repository info. Started on small persistence implementation.

2009-08-20 Th Further setup (Groovy, Groovy-Plugin), starting on small crawler in Groovy to crawl websites for code repos.

2009-08-17 Tu More installing, setting up Groovy to do some testing, first steps with Groovy

2009-08-16 Mo Installed workplace, set up stuff, read papers.