Mailing List Archives
Authenticated access
|
|
|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] condor, sqlite, python
- Date: Mon, 5 Apr 2010 07:54:39 -0400
- From: Mag Gam <magawake@xxxxxxxxx>
- Subject: Re: [Condor-users] condor, sqlite, python
Brian,
Really depends on what how many times you are doing this. If you need
to do this several million times then condor maybe the right tool for
you.
I also suggest you moving away from sqlite when working with "large"
data because its really not efficient. Perhaps have your data in a tab
or ccommand seperated values which will make data manipulation much
easier.
You should also look into the UNIX join command it can do many things
to avoid a database.
If you can show us some same data it would be easy to create a sample
script or even a condor job...
hope this helps!
On Thu, Apr 1, 2010 at 12:21 PM, Geary, Brian W. <bgeary@xxxxxxxxx> wrote:
> Hello.
>
>
>
> We are beginning a research project which will by joining 2 million rows
> against 250,000 rows three times.
>
> We can either do the 3 joins in one query or do three separate queries with
> 1 join.
>
>
>
> We are using sqlite and python to execute the sql queries and there is a bit
> of logic in the python as well.
>
> We can use Windows or *nix for this project or both.
>
>
>
> Can someone explain if Condor would be appropriate for this research
> project? Would we just submit the
>
> php script as a job or some other scenario.
>
>
>
> Thanks..
>
>
>
>
>
> --
>
> Brian
>
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
>