I wrote simple code to read a csv file consisting of 107085 lines using Readline function after I loaded the entire file in Memory. it consists of lines like
"105110";"217.110.197.72";"217.110.197.95";"3647915336";"3647915359";"DE";"Germany"
and it could take over 10/15mins easy to find something...the speed was insanely slow...
yet mysql can search through this thing in milliseconds, it's absolutely incredible, I think one day i'm going to take a look at how it searches.I thought about multiple threads each being assigned a certain number of lines to search, but it still wouldn't = mysql's incredible speed.
A simple word: "index".
Databases do not scan linearly unless asked on purpose.
Instead they search a binary tree or a hash or a combination stored into an "index file".
Even if there is no index file, it should not take longer than, say, 15 milliseconds to find something in a file with 100,000 lines.
I really want to see what kind of code takes 15 min to find something in a 100 000+ lines file :green