Data Stage Logging in 8.1

We now have an enviornment set up and a few of us building flows using designer 8.1. At some point, you realize logging becoming a real issue. As in the job completes but the data is still being logged. Interesting find was that 8.1 uses the db to do the logging and everthing is stored in their xmeta database. so with multiple jobs and multiple users it quickly becomes a bottleneck. To make matters worse, a few of these tables seems like are temporary with no indexes.  Fortunately you have a way to turn back to has file logging. You can do this by switching rtlogging and orlogging.

The project specific DSParams has the rtlogging and orlogging parameters. you need to set RTLogging to 1 and ORLogging to 0

 

Some errors we used to see because of performance of DB logging -

When I tried to log in I got the following message:

ACCESS DENIED: Project currently locked.

The project is currently locked by the Administrator. Try again later or see your administrator.

 additionally we would see the director just freeze up as we try to open the logs

you could not rerun a job for a very long time after it was done,

Job is not in a runnable state. this we learnt was because the logs were still being written long after job completed and the server thought the job is still active.

Choosing an ETL tool

We started off on our process of selecting an ETL tool.  This tool would replace our inhouse java based data flow engine to load of Tlog and other large retail data.

The evaluation process was geared towards measuring performance, scalability and the ability to quickly build configurable flows that works across customers. After the first round of evaluation we were left with the big 3, Ab Initio, Ascential Data Stage from IBM and Informatica.

We gave each of the vendors two of our current flows to implement in their tool in a week. We ranked them as AI, Data Stage, Informatica. While IBM was a not exactly a close second when it came to performance, with sufficient hardware, we felt we could get what we needed in terms of performance. They also seemed to have some cool features w.r.t to integrating to web applications etc.  of course closer to our budget which is equally important in these times. so Ascential Data Stage it is…