iddqd: 2009

Sunday, December 20, 2009

TDD Test Driven Development

Firmy szaleją na punkcie AGILE. W mojej firmie na przykład wprowadzono SCRUMa. Jestem DBA i widzę skutki działania programistów i czas jaki poświęcają na daną funkcjonalność oraz UTRZYMANIE.

Fundamenty projektu. Początkowy etap rozwoju jest "najprostszy". Wszyscy są podekscytowani, programista ma coś nowego i 80% projektu powstaje bardzo szybko.
Dopieszczanie projektu. Mamy fundamenty, wstawione okna i podłączony prąd (nie ma zamontowanych włączników tylko gołe druty), w łazience czeka najlepsze jacuzzi na świecie. Wykończeniowcy mozolnie dopieszczają nowy dom.
Wejście mieszkańców. Horror, w jednym miejscu ktoś zapomniał wstawić włącznika i lokator dotknął dłonią przewodów pod napięciem, część włączników w ogóle nie działa. Okazało się, że rury są nieszczelne i woda wycieka ze ściany a nie leci do jacuzzi. Jesienią i zimą podwórze zamienia się w bagno bo studzienki nie są w stanie odprowadzić wody.
Poprawki. Zrobiliśmy brakujący włącznik skuliśmy ściany i wymieniliśmy nie działającą część instalacji elektrycznej , poszerzyliśmy studzienki kanalizacyjne, nawet uszczelniliśmy rury. Okazało się jednak, że po uszczelnieniu rurociągu trzeba zmienić rury kanalizacyjne bo nie dają rady z odprowadzaniem zużytej wody. Podczas kucia uszkodziliśmy kable ethernetowe i przestał działać internet oraz system alarmowy oraz działająca do tej pory część instalacji elektrycznej. Zleceniodawca zerwał umowę i musimy płacić karę umowną.

Tak dokładnie wyglądają projekty informatyczne w dziewięćdziesięciu procentach wypadków. Na ratunek przyszły zwinne metody prowadzenia projektów. Niestety to nie jest panaceum na problemy. Scrum wymaga dobrych i doświadczonych programistów w projekcie. Programistów umiejących pisać testy.

Bez pisania testów nie da się wykorzystywać Scruma. Na dobrą sprawę młyn bez testów spowalnia tworzenie oprogramowania ponieważ iteracje często wymagają gwałtownych przeróbek i ręczne wyszukiwanie implikacji zmian ( nowych błędów ) to mordęga.

Dobry programista projektuje przed startem implementacji. Implementacje zaczyna od pisania testów. Nie da się przewidzieć wszystkich przypadków ale unika się kompromitujących wpadek.

Scrum jest prosty ale dla doświadczonych programistów.

http://en.wikipedia.org/wiki/Test-driven_development

Friday, December 4, 2009

Django Pisa (polskie czcionki)

I use Pisa to generate reports. I had problem with polish lettersm solution is to replace included fonts (arial, helvetica) with own.
Copy fonts to /project/directory/site_media/font.

#-*- coding: utf-8 -*- 
import cStringIO as StringIO

import ho.pisa as pisa

from django.contrib.auth.decorators import login_required
from django.http import HttpResponse
from django.shortcuts import get_object_or_404
from django.template import Context
from django.template.loader import get_template
from django.utils.translation import ugettext_lazy as _

from arap.report.models import Review
from arap.report.models import Type
from arap.settings import  MEDIA_ROOT

@login_required
def print_report(request, id):
    """
    Generuje raport z przeglądu.
    """
    template = get_template('report/print.html')
    review = get_object_or_404(Review, id=id)
    context = Context({'pagesize':'A4', 'review':review, 'MEDIA_ROOT':MEDIA_ROOT})
    html  = template.render(context)
    result = StringIO.StringIO()
    pdf = pisa.pisaDocument(StringIO.StringIO(html.encode('UTF-8')), result)
    if not pdf.err:
        response = HttpResponse( result.getvalue() )
        
        response['Content-Type'] = 'application/pdf'
        response['Content-Disposition'] = 'attachment; filename="%s-%s-%s.pdf";' \
            %(review.server,review.report_type, review.created)
        return response
    return HttpResponse(u"Coś nie bangla.", mimetype='text/html')

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 
    "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head> 
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

<style type="text/css"> 
@font-face {
   font-family: "arial","helvetica","symbol";
   src: url({{ MEDIA_ROOT}}/font/arial.ttf);
}

@font-face {
   font-family: "arial black";
   src: url({{ MEDIA_ROOT}}/font/arialbd.ttf);
}

@font-face {
   font-family: "verdana";
   src: url({{ MEDIA_ROOT}}/font/verdana.ttf);
} 


@font-face {
   font-family: "Times New Roman" ,"wingdings";
   src: url({{ MEDIA_ROOT}}/font/Times_New_Roman.ttf);
}

@font-face {
   font-family: "Comic Sans Ms";
   src: url({{ MEDIA_ROOT}}/font/Comic_Sans_MS.ttf);
}

@page {
  margin: 1cm;
  margin-bottom: 2.5cm;
  @frame footer {
    -pdf-frame-content: footerContent;
    bottom: 2cm;
    margin-left: 1cm;
    margin-right: 1cm;
    height: 1cm;
  }
} 

html, body, table, caption, tbody, tfoot, thead, tr, th, td {
    font-family: "Times New Roman";
}
</style>

<title>Raport {{ review.report_type }} {{ review.server }} {{ review.created }}</title>
</head>
<body> 
  {{ review.report|safe }} 
  <div id="footerContent">
    {%block pager %}
      <pdf:pagenumber>
    {%endblock%}
  </div>
</body>
</html>

Saturday, November 7, 2009

How to find locks in postgres ?

Table pg_locks is very usefull...

select
 pg_stat_activity.datname,
 pg_class.relname,
 pg_locks.transaction,
 pg_locks.mode,
 pg_locks.granted,
 pg_stat_activity.usename,
 substr(pg_stat_activity.current_query,1,30),
 pg_stat_activity.query_start,
 age(now(),pg_stat_activity.query_start) as "age",
 pg_stat_activity.procpid
from 
 pg_stat_activity,
 pg_locks left outer join pg_class on (pg_locks.relation = pg_class.oid)  
 where
 pg_locks.pid=pg_stat_activity.procpid order by query_start;

Friday, November 6, 2009

Update massive number of records

Simple Pl/SQL procedure which help to do massive update.

create or replace PROCEDURE move_column AS
  type num_table IS TABLE OF NUMBER INDEX BY pls_integer;
  round_trips number;
  recid_table num_table;
  avid_table num_table;
  scale NUMBER;
  starting_point NUMBER;
  ending_point NUMBER;
BEGIN
  SELECT recid,
    attrib_value_id bulk collect
  INTO recid_table,
    avid_table
  FROM very_big_table;
  scale := SQL % rowcount;

  round_trips := TRUNC((scale + 9999) / 10000);


  FOR bite IN 1 .. round_trips
  LOOP
    starting_point :=(bite -1) *10000 + 1;
    ending_point := least(scale,   bite *10000);
    forall this IN starting_point .. ending_point

      UPDATE another_very_big_table SET rec_id = recid_table(this)
       WHERE id = avid_table(this);
    COMMIT;
    --DBMS_OUTPUT.PUT_LINE('Move: ' || ending_point);
  END LOOP;

END move_column;

Wednesday, November 4, 2009

read write file from gridfs

This simple program reads file from gridfs and write its content to local file system.

//gridfs_to_file.cpp
#include <iostream>
#include <vector>
#include <fstream>

#include <boost/algorithm/string.hpp>

#include <mongo/client/dbclient.h>
#include <mongo/client/gridfs.h>

// g++ gridfs_to_file.cpp -lmongoclient -lboost_thread -lboost_filesystem -o gridfs_to_file

using namespace std;
using namespace mongo;

int main(int argc, const char **argv) {
   const char *gridFileName = "";
   std::fstream out;


   if (argc != 3) {
      cerr << "Usage " << argv[0] << " gridfs_file local_file"  << endl;
      return -12;
   }

   out.open(argv[2], ios::out);

   if ( !out.is_open()) {
      cerr << "Can't open " << argv[2] << endl;
      return -2;
   }
   gridFileName = argv[1];

   DBClientConnection c;
   c.connect("localhost"); 
   cout << "connected ok" <<endl;
   GridFS gfs = GridFS(c, "test", "testcpp");
   GridFile gf = gfs.findFile(gridFileName);

   if (true != gf.exists()) {
      cerr << "There is no file like " << argv[1] << endl;
      return -2;
   }
   gf.write(out);   
   out.close();

   return 0;
}

This program reads file from local file system and send it to gridfs.

#include <mongo/client/dbclient.h>
#include <mongo/client/gridfs.h>

// g++ local_to_gridfs.cpp -lmongoclient -lboost_thread -lboost_filesystem -o local_to_gridfs

using namespace std;
using namespace mongo;

int main(int argc, const char **argv) { 
    char *data;
    int len;

    if (argc != 3) {
        cerr << "Usage " << argv[0] << " localFile remoteName "  << endl; 
        return -12;
    }

    fstream in(argv[1], ios::in|ios::binary|ios::ate);
    
    DBClientConnection c;
    c.connect("localhost"); 
    cout << "connected ok" <<endl;

    GridFS gfs = GridFS(c, "test", "testcpp");
    if ( in.is_open() ) {
        len = in.tellg();
        in.seekg( 0, ios::beg);
        data = new char[len];
        in.read(data, len);
    }
    //storing file

    gfs.storeFile(data, len, argv[2], "text/plain");
    delete [] data;
    cout << "file stored to gridfs" << endl;

    return 0;
}

Tuesday, November 3, 2009

Great news for MongoDB users

When I saw Mongodb, it was most promising non-relational database. There's no time to lose.

"10gen, a New York-based developer of an open-source, non-relational database called MongoDB, has raised $3.4 million in Series B funding. Flybridge Capital Partners led the round, and was joined by return backer Union Square Ventures."

http://www.pehub.com/54541/mongodb-developer-10gen-raises-34-million/

Friday, October 23, 2009

simple program to save file from c++ to (mongodb) gridfs

MongoDB C++ Tutorial http://www.mongodb.org/pages/viewpage.action?pageId=133415

The best start is building MongoDB from source (Ubuntu|Debian).

$sudo apt-get install g++ scons libpcre++-dev libmozjs-dev libpcap-dev libboost-dev
$cd /usr/src
$sudo git clone git://github.com/mongodb/mongo.git
$cd mongodb
$scons
$scons --prefix=/opt/mongo install
$cd ~
$gvim test_gridfs.cpp

#include <iostream> 
#include <vector>

#include <boost/algorithm/string.hpp>

#include <mongo/client/dbclient.h>
#include <mongo/client/gridfs.h>

// g++ tutorial.cpp -lmongoclient -lboost_thread -lboost_filesystem -o tutorial

using namespace std;
using namespace mongo;

int main(int argc, const char **argv) { 
const char *fileName = "";
std::vector<std::string> strs;

if (argc != 2) {
cerr << "Usage " << argv[0] << " local_file "  << endl; 
      return -12;
   }

   fileName = argv[1];
   //to generate gridfs file name
   boost::split(strs, fileName, boost::is_any_of("/"));

   DBClientConnection c;
   c.connect("localhost"); 
   cout << "connected ok" <<endl;

   GridFS gfs = GridFS(c, "test", "testcpp");
   gfs.storeFile(fileName, strs[strs.size()-1]);
   cout << "file stored" << endl;

   return 0;
}

g++ -o file_to_gridfs.o -c -I/opt/mongo/include file_to_gridfs.cpp
g++ -o file_to_gridfs file_to_gridfs.o -L/opt/mongo/lib -lmongoclient -lboost_thread -lboost_filesystem

Tuesday, October 20, 2009

mongoDB gridfs and sharding

If you want use gridfs and sharding chunks. In example is the mistake. Sharding by "_id" dosen't work.

Working example.

First ....
http://www.mongodb.org/display/DOCS/A+Sample+Configuration+Session

and then ...
$mongo
> use admin
switched to db admin
> db.runCommand( { shardcollection : "test.dexters.chunks", key : { n : 1 } } )
{"collectionsharded" : "test.dexters.chunks" , "ok" : 1}

$vim test_load.py
====================================================
#!/usr/bin/env python
import sys
import os.path
from pymongo.connection import Connection
from gridfs import GridFS

connection = Connection("localhost", 27017)
db = connection["test"]

name = os.path.basename(sys.argv[1])

fs = GridFS(db)
fp = fs.open(name, 'w', 'dexters')

for l in open(sys.argv[1]):
    fp.write(l)
fp.close()
====================================================

There is at leas two sharding strategy, by file_id or by (n, id).
Sharding by file_id is no RAID or RAID-1 (on file level) sharding by n when you use many servers can be like (RAID-0, RAID-10). On collection level performance is always RAID-0, RAID-10 ;-)

Monday, October 19, 2009

Problem with Oracle Enterprise manager

If you can't run Oracle Enterprise manager ...


$ ./opmnctl startall

ias-component/process-type/process-set:
HTTP_Server/HTTP_Server/HTTP_Server/

Error
--> Process (index=1,uid=186798,pid=9999)
failed to start a managed process after the maximum retry limit

$less Log:
/usr/home/oracle/product/10.1.3/OracleAS_1/opmn/logs/HTTP_Server~1

ias-component/process-type/process-set:
OC4J/home/default_group/

Error
--> Process (index=1,uid=1867980864,pid=450578)
time out while waiting for a managed process to start
Log:
/usr/home/oracle/product/10.1.3/OracleAS_1/opmn/logs/OC4J~home~default_group~1

You must chenge directory to Apache log directory an delete all files.


$cd ../../Apache/Apache/logs
$rm *

That's all ...

Friday, October 16, 2009

How to drop all tables from database using one SQL query

How to drop all tables from database using one SQL query without destroying database. It's impossible in Postgresql, MySQL ,Oracle. But we can generate sql.

#MySQL
mysql> SELECT CONCAT('DROP TABLE ', TABLE_NAME,' ;') FROM information_schema.TABLES WHERE TABLE_SCHEMA='mysql';
+----------------------------------------+
| CONCAT('DROP TABLE ', TABLE_NAME,' ;') |
+----------------------------------------+
| DROP TABLE columns_priv ;              | 
| DROP TABLE db ;                        | 
| DROP TABLE func ;                      | 
| DROP TABLE help_category ;             | 
| DROP TABLE help_keyword ;              | 
| DROP TABLE help_relation ;             | 
| DROP TABLE help_topic ;                | 
| DROP TABLE host ;                      | 
| DROP TABLE proc ;                      | 
| DROP TABLE procs_priv ;                | 
| DROP TABLE tables_priv ;               | 
| DROP TABLE time_zone ;                 | 
| DROP TABLE time_zone_leap_second ;     | 
| DROP TABLE time_zone_name ;            | 
| DROP TABLE time_zone_transition ;      | 
| DROP TABLE time_zone_transition_type ; | 
| DROP TABLE user ;                      | 
+----------------------------------------+
17 rows in set (0.02 sec)
Execute drop commands many times becouse of foreign keys ;-)

#Postgresql
SELECT 
'DROP TABLE '||c.relname ||' CASCADE;'
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_roles r ON r.oid = c.relowner
LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind IN ('r','')
AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 1;

?column?                     
-------------------------------------------------
DROP TABLE auth_group_permissions CASCADE;
DROP TABLE auth_group CASCADE;
DROP TABLE auth_message CASCADE;
DROP TABLE auth_permission CASCADE;
DROP TABLE auth_user_groups CASCADE;
...

#Oracle
SELECT 'DROP TABLE '||table_name||';' FROM user_tables;
'DROPTABLE'||TABLE_NAME||';'               
------------------------------------------ 
DROP TABLE SYS_EXPORT_SCHEMA_01;           
DROP TABLE MON_NO_CHECK;                   
DROP TABLE REWRITE_TABLE;

RoR sucks

I host my websites on megiteam.pl. I'ts the best Python, Ruby hosting in the world in this prize. I tried to use mephisto,typo for this blog... what a mistake ...

My big application write in Python (Django) on load(400 req/s) use 20MB per child.
Mephisto,Typo after restart use 41MB to 60MB. I've tested Typo on my laptop (5 req/s) 120MB per child.


24306 *     41.46MB /usr/bin/ruby1.8 /var/lib/gems/1.8/bin/thin -a 127.0.0.1 -p 10600 -e production start

mongoDB rocks

I have found this project very useful for big sites and heavy load.

mongodb, comparision.

vim ./test_load.py
======================================
#!/usr/bin/env python
from pymongo.connection import Connection

connection = Connection("localhost", 27017)
db = connection["test"]
for collection in db.collection_names():
   print collection

collection = db["testCollection"]

for i in range(10000):
   collection.insert({"i":i, "body": 'a - %s'%(i,)})
======================================
$time ./test_load.py
real 0m5.301s
user 0m4.160s
sys 0m0.230s
$du -hs /data/db/
81M /data/db/

iddqd