Professional staff - рабочие заметки

Converting Cyrillic ID3 tags

2011-07-02T14:47:00.001-07:00

find . -name '*.mp3' -print0 | xargs -0 mid3iconv -eCP1251 --remove-v1

The three layers of GenABEL

2011-05-22T10:31:00.000-07:00

I think there is a growing confusion about GenABEL, and I start getting questions like: What the hell is it? To whom does it belong? Is it free? Will it stay free? Is it supposed to make money? What happens next? Also the rumors that I am leaving Academia spread, building some anxiety.

To start with, please keep in mind that whatever happens, I am dedicated to the free GenABEL in the open source spirit. The GenABEL is and will stay open (re)source: free of charge to anyone, open source, modifiable, and redistributable.

Ok, now to business: To start with, and may be to confuse you even more, there is no such thing as "the GenABEL". Actually, there are three things – the GenABEL package, the GenABEL suite, and the GenABEL project. The GenABEL project includes the suite, which includes the package. I will elaborate below on each of them in turn, starting with smaller entities.

*** The GenABEL package

What is it: This is free open source package for R (R is also, a free, open source software).

To whom does it belong: It is released under GNU GPL, so the answer is it free to copy, modify, and redistribute to everyone. It is "mine" in the sense that I feel responsible to keep it in good shape. At the same time, many other people contributed the ideas, code, documentation, suggestions, etc. to it. In that, the package belongs to the community. In the release 1.6-7 of the package, I have removed my name from the DESCRIPTION file and the “GenABEL developers” are credited as authors. You can see who has developed specific function, and what paper should be referenced, by looking at the man page of a function.

Will it stay free: Sure. It is released under GNU GPL, so the source cannot be closed or charged money for. Moreover, many people contributed; guess who can claim the ownership? I am feeling quite happy about this.

Is it supposed to make money: Not directly. The code is available to everyone for free (provided any derivative is also made available for free). This does not exclude indirect uses, e.g. if a company approaches me with a question on how to use the GenABEL package most effectively, I may consider charging them for consultancy. The same goes for the cases when I am invited to teach a course, or help a company developing a software making use of the package. If someone wants to have specific functionality extended within GenABEL package, and is willing to pay money for it, I will not say ‘no’ to get some extra income. Note that according to GNU GPL whatever is developed must become a part of the free GenABEL package if the GenABEL code was used!

What next: There are many plans about GenABEL; after 4 years, it is still (an will remain) under active development. During 2010, we had 10 public (CRAN) releases adding and improving much if the functionality.

*** The GenABEL suite

What is it: This is a set free open source packages, mostly for R. However, the set also includes standalone applications (e.g. ProbABEL) and libraries (e.g. filevector). The GenABEL package is part of the GenABEL suite.

To whom does it belong: Right now, all packages are released under GNU GPL, so the first answer is that the “GenABEL suite” is free to copy, modify and redistribute. At the same time, each individual package ‘belongs’ to individuals, who contributed these packages.

Will it stay free: Sure. My position is that only the packages, which are open source, free to use, modify, and redistribute (recursively) should be in the GenABEL suite.

Is it supposed to make money: Not directly, but there may be many indirect uses. For example, a company may package GenABEL suite with a cloud computing (or whatsoever), and sell the computations as a service. No complains here.

What next: My sincere hope is that the GenABEL suite will grow both in quality and quantity and will become a leading software suite for statistical analyses of polymorphic genomes.

*** The GenABEL project

What is it: this is a framework for methodology development, based on an idea of methodology development as a three-stage process including methodology development itself (math), implementation of this methodology in a usable software (the GenABEL suite), and application of the software to solve real-life problems.

To whom does it belong: this is a community effort, and it belongs to the community. While I am currently the coordinator and in that have most influence and control, this can (and should) change.

Will it stay free: Sure. The GenABEL project is a kind of a workshop. If people are not satisfied with a setting of a workshop, they just organize alternative workshop.

Is it supposed to make money: Not directly. I must admit I can also hardly imagine how it can make money indirectly as well. See remarks about the suite and package for more info.

What next: My sincere hope is that in the GenABEL project framework, we will work out new standards for open methodology development and open manuscript writing; will formulate most important (and fun) problems of statistical genomics and come up with great new solutions to these; develop excellent software; will publish top papers; will apply our software to solve burning problems of biology; and will contribute to public good.

GenABEL 1.6-5 released!

2011-02-08T09:49:00.000-08:00

Here is the post on forum:

http://forum.genabel.org/viewtopic.php?f=6&t=2#p41

Multiple pages?..

2011-01-24T07:45:00.000-08:00

I have posted that finally I could get nice HTML out of Rnw (tex). Here is a little problem though -- we would like to have all-in-one-page (this is what tth will do for us), but also in latex2html-style, i.e. every subsection is a separate HTML.

Any ideas how to do that?

Converting Sweave Rnw to HTML

2011-01-24T04:41:00.000-08:00

It took loooooooooong time to figure out how we can convert all these huge Rnw to HTML, but apparently this is the way to go:

Install TtH
R CMD Sweave myRnwfile
sweave2html mytexfile

http://biostat.mc.vanderbilt.edu/wiki/Main/SweaveConvert

It says some words about handling graphics, but I rather commented graphics-related lines (lines 2-4) in the sweave2html script and things go fine with graphics.

Get him out of the way!

2010-10-22T11:12:00.000-07:00

I am going to submit a grant application for *ABEL development. To justify the need for that, I did a lot of things -- in particular, I've analyzed the e-mails I was receiving and sending about *ABEL: you know, people are asking how you do this and that analyzes, and you reply.

I have discovered some figures which seemed impressive to me. The first CRAN release of GenABEL came out in Feb. 2007. In March, 2007, I got only few questions. At the moment, the number of e-mails I receive about *ABEL per month is about 200. In total, since the first CRAN release, I got about 3,000 e-mails about *ABEL. About half of them I had and did reply to.

Anyways. I drew the number of *ABEL mails per month -- see the graph below

... and was puzzled about irregular pattern. So I tried to figure out what is going on -- and managed to identify the events (marked with red and green circles) which apparently were driving the pattern. I was very happy I could attribute a lot of variation to some factors :)

So, in this happy scientific state I dropped by Lennart's office to show the graph. See, it starts with few e-mail in March 2007, ... blah, blha, blah, ... the pattern ... But I managed to identify the critical events! So -- do you Lennart want to know what are the green dots -- what are the events which boost the number of e-mails up? Surely he did want to know. So I tell -- feeling myself as some prophet -- THESE ARE LAUNCHES OF NEW PRODUCTS. Around Jan 2009 (green upper point #1) we launched ProbABEL. Around Jan. 2010 (green upper point #2) we launched DatABEL and MixABEL development. Wow! -- was Lennart's reply; -- but what about the deeps -- the lower red points? It seems that there is some obstacle we got in the user support model he was saying. Apparently what happens is that a number of e-mails get unanswered, so the flow is broken. And what is funny, he continued, is that this obstacle got a kind of 6-months period -- it comes into play in June/July and December/January (see the figure). We need to figure that out and get that out of our way! -- he said.

Oh yes, tell I, feeling very smart. I have figured that factor out! These red dots apparently coincide with my vacations. So, you see, what is happening is that I get a number of e-mails, but if I am on vacations, they are not answered and are not discussed any... At this point I got suddenly interrupted by Lennart who shouted -- so YOU are the obstacle!.. We got to get you out of the way!

%&^$^$!!!! (!) &*#%(!!

... after all, I am a scientist. I am statistical geneticist. I do respect the numbers. So, thinking realistically, Lennart is totally right. What we need to do to solve the user support problem is to get me out of the *ABEL way. Now all project communications goes through me. And this is not right. There are many guys around who can answer the questions -- why not letting them answer directly? Moreover, as project is already quite big, there are many parts and questions where other guys can give a better answer then me! So, right, let's do that -- get Yurii out of the way!

What we think of doing is establishing totally different user support model. It used to be all-requests-go-through-Yurii model, now we gonna make that a community model. There are many guys around who are very expert and contributed a lot already. It is very happy that GenABEL already got a very large number of expert volunteers participating in the development. Current idea is that we will set up a mailing-list/forum/??? for user questions and for developers, so the information can flow freely. We will update the web-site and provide the mechanism for the people to join in and contribute to documentation and tutorials. Any suggestions are welcome!

Русификация Blackberry Bold

2009-08-16T09:23:00.001-07:00

http://4pda.ru/forum/index.php?showtopic=98141&st=0

berkeley db env vars

2009-08-13T13:24:00.000-07:00

Some influential environment variables:
CC C compiler command
CFLAGS C compiler flags
LDFLAGS linker flags, e.g. -L if you have libraries in a
nonstandard directory
LIBS libraries to pass to the linker, e.g. -l
CPPFLAGS C/C++/Objective C preprocessor flags, e.g. -I if
you have headers in a nonstandard directory
CXX C++ compiler command
CXXFLAGS C++ compiler flags
CXXCPP C++ preprocessor
CPP C preprocessor
F77 Fortran 77 compiler command
FFLAGS Fortran 77 compiler flags

how to link to bdb

2009-08-13T08:58:00.000-07:00

Libraries have been installed in:
/usr/local/BerkeleyDB.4.7/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- use the `-RLIBDIR' linker flag

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

Установка perl-модулей с CPAN

2009-08-13T07:14:00.001-07:00

всегда забываю как устанавливать модули с CPAN. Напоминалка:

perl -MCPAN -e 'shell'
...
install PACKAGENAME

Long long long ...

2009-07-28T02:44:00.001-07:00

Вот такая вот ошибка:

test_data_types.cpp: In function 'int main()':
test_data_types.cpp:11: error: 'long long long' is too long for GCC

ssh login without password

2009-07-25T13:04:00.000-07:00

Нужно копировать огромные файлы по scp; копирование должно продолжаться в backgroung, после того как я отлогинился. Помогло "SSH login without password"

Follow-up on "Builing 64bit R & Bioconductor on Solaris 10"

2009-07-25T08:45:00.000-07:00

When building R for Solaris following instructions from "Builing 64bit R & Bioconductor on Solaris 10" (see "R для Solaris 10 на AMD64"), GenABEL failed to compile. Maksim figured out that we need to change the "env" file (and re-build R); the 3rd line changes to

export CXX="CC -m64 -lCrun -lc -lCstd"

Many many thanks Maksim, great job!

Reading big files using ifstream::read

2009-07-25T08:37:00.000-07:00

I am reading very big files using fstream::read. I first read to my internal cache using command

object.read((char*)char_buffer,current_cache_size)

When trying cache > 2048 Mb, the program is stuck.

OK, the trick is that

istream& read ( char* s, streamsize n );

where

"streamsize" is an implementation-defined synonym of one of the signed basic integral types (generally signed int or signed long). So I am apparently having "signed int" implementation:

2^(32-1) == 2048*1024*1024

So I have to read in smaller blocks and then fill up my cache...

Replacing class checks

2009-07-22T14:51:00.000-07:00

GenABEL's checks for passed objects class did not comply with S4 (thanks to Maxime Rotival who noticed that). So I had to replace all checks like

if (class(a)=="xxx") ...
if (class(a)!="xxx") ...

To

if (is(a,"x")) ...
if (!is(a,"x")) ...

Vim syntax:

:s/class *( *$[a-zA-Z.-]\+$ *) *== *"$[a-zA-Z.-]\+$"/is(\1,"\2")/g
:s/class *( *$[a-zA-Z.-]\+$ *) *!= *"$[a-zA-Z-.]\+$"/!is(\1,"\2")/g

Sed syntax:

sed -e 's/class *( *$[a-zA-Z.-]*$ *) *== *"$[a-zA-Z.-]*$"/is(\1,"\2")/g' ...
sed -e 's/class *( *$[a-zA-Z.-]*$ *) *!= *"$[a-zA-Z.-]*$"/!is(\1,"\2")/g' ...

(note that here you need to have "-" last!)

Perl script to replace that in all files:


@files = `ls`;
foreach (@files) {
     chomp;
     `sed -e 's/class *( *\\([a-zA-Z.-]*\\) *) *== *"\\([a-zA-Z.-]*\\)"/is(\\1,"\\2")/g' $_ > $_.tmp1`;
     `sed -e 's/class *( *\\([a-zA-Z.-]*\\) *) *!= *"\\([a-zA-Z.-]*\\)"/!is(\\1,"\\2")/g' $_.tmp1 > $_.tmp2`;
     `mv $_.tmp2 $_; rm $_.tmp1 $_.tmp2`;
}

(I know it is not really perl, but it works -- any suggestions for a nicer way to do that are welcome)

Question: can one think of regexp which will do that in a single match?

R для Solaris 10 на AMD64

2009-07-22T14:40:00.001-07:00

Собрал R для Solaris 10 на AMD64. Очень помогло: Builing 64bit R & Bioconductor on Solaris 10. Однако пакеты не все компилятся пока... В частности qvalue требует tcltk -- установка в системе не помогает...