hello and welcome to my first question

[pianodervish]

hey guys!
I just started assembly about 2 months ago (its what we are learning in a computer systems course at my school). I was wondering what is the fastest way of generating an insanely huge matrix?
It would be on the order of 22,000 x 22,000.
My program needs to solve an insanely huge sudoku-like problem. Think of it as a blank matrix that i need to solve. I only have the has the sum of each row, column, and diagonal (both ways). If i have 22,000 rows and columns I have a list of 22,000 x 4 sums and my output has 484,000,000 (half a trillion) numbers. Im very proficient in C  so if its easier to do in C, then so be it.

note: i cant use sparse matrices because I have no idea whether the matrix will contain a lot of zeros or not.
The output (matrix) can be a stream of data sent to some hard drive, but the main problem is that the algorithms used to solve these problems need partial solutions in the matrix during runtime. If this is impossible, then what is the best and fastest way to generate the largest matrix possible?

Thanx, David

[jj2007]

Hi David,
Welcome to the forum.
If you are talking dwords (or real4), you need 2 GB of free RAM to get reasonable speed. Memory mapping of files is something you may be looking for.
The creation of the matrix itself is not the problem. VirtualAlloc 22000*22000*4 should do the job. The interesting part is after that... tell us more.

[pianodervish]

Thanx for the answer! Ill definately look into it.

Ok well this is part of a project. Basically, we have a new camera technology that can reach super high resolutions through mostly computation. Sounds crazy right? Well imagine light shining through a window. Now imagine a piece of cardboard over that window. The light stops shining right? Well lets say you move the cardboard back and lets define 5 steps back as allowing the light to completely shine through. That means, with one step back, you increase the amount of light shining through by 20%, since 5 steps is 100%.
Now, if you do this moving the carboard up and down (for the y axis) side to side (for the x axis) rotated 45 degress to the right (z axis) and 45 degress to the left (L axis) you end up with "lines of light" and there appropriate sums.
Computationally, its like knowing the sum of the diagonals, row, and columns of this huge matrix. Each value in the matrix corresponds to a pixel value. When we solve this matrix, it gives us a picture. Seems crazy that just from light levels you can make a picture right? But thats why the matrix is so large, since there can only be one solution.
The actual camera uses nano dots and other things to get this amount of data insanely fast. If we can get this too work, we might get rid of the word megapixel altogether.

Our main problem right now is getting this data to the computer in an efficient way and then manipulating that data. My job is to come up with a bunch of algorithms. We have 3 prototypes but are using a brute force algorithm to solve this matrix which takes about a minute. We want to get this down to seconds.

[pianodervish]

To further illuminate the matrix problem, imagine you have this matrix with these sums:
1|__|__|
3|__|__|
   3    1

a solution would be:
1   0
2   1

Imagine this with a huge matrix and the addition diagonal info.

[dedndave]

the size of each data element seems critical
but, i would think you would want to perform the matrix conversion inside the camera, prior to storing the image

[pianodervish]

true but we need the computational speed of the computer to recreate the image/matrix in the first place no? Though im not on the eecs side of this project, im guessing the camera will stream the data to the computer, and when it comes the end of a sequence, send some binary sequence that represents the end of the scanning. So, itll stream 22,000 sums for lets say the x axis, then send something which tells the computer that its done with the x and moving to the y.
in the future we prob would move to somehow porting this to a completely portable solution but for now we are mostly interested in optimizing our program. think of the current camera as something like a webcam: completely useless without a computer

[edit] But its not like a webcam in the sense that the computer doesnt have to do heavy computation on the streaming data before creating the picture