Is there a way to capture input/output data streams during an Internet connection in an
assembly application?
Here is what I would like to do. My email client, which is MS Outlook Express connects to
Microsoft's POP3 server (pop3.email.msn.com on port 110) every 3 minutes to check for new mail.
I would like to see what Outlook Express is sending to the POP3 server and also see what the
POP3 server is responding back. I am trying to create my own POP3 client and I know the general
basic protocol but every time I try to connect to the server with my application I always get
an -ERR response from the server. Here is what I am doing (I am the client):
1) Client: Connect to Microsoft's POP3 server on port 110
2) Server: +OK BAY0-POP04 POP3 Server ready
3) Client: USER my_user_name
4) Server: -ERR command not implemented
Now, I have another POP3 server that I use also:
1) Client: Connect to xanthus.host4u.net
2) Server: +OK POP3 xanthus.host4u.net v2001.78rh server ready
3) Client: USER my_user_name
4) Server: +OK User name accepted, password please
5) Client: PASS my_password
6) Server: +OK Mailbox open, 5 messages
So, my POP3 program works for one server but not for the other. If I could write a program that
captures the i/o from Outlook to Microsoft's server I could then see what command strings are
being used. It can't be a mystery because other POP3 clients (other than Outlook) are able to
connect and get through.
Here's the pop3 RFC: http://www.faqs.org/rfcs/rfc1939.html
By the look of it, your client works fine but the server doesn't. What server are you talking about for the first thing? It's odd that a server would report that it doesn't support the USER command.
Robert,
Go to Japheth's website, http://www.japheth.de/, and download Joe.exe
This program is a mailreader (very nice, actually). When you run this program, you will see a dialog window open with the mail server and it will show you all the commands and replies.
Paul
Quote from: sheep on January 31, 2005, 06:29:27 PM
By the look of it, your client works fine but the server doesn't. What server are you talking about for the first thing? It's odd that a server would report that it doesn't support the USER command.
In the sample that I posted it stated that it is 'pop3.email.msn.com' I know the server must work because Outlook Express uses that server to retrieve my email from the msn email data base. Also, if I use another email client (such as Endura or Opera) and I give them the name of the server and my user name + password they are able to retrieve my email.
Quote from: pbrennick on January 31, 2005, 06:44:14 PM
Robert,
Go to Japheth's website, http://www.japheth.de/, and download Joe.exe
This program is a mailreader (very nice, actually). When you run this program, you will see a dialog window open with the mail server and it will show you all the commands and replies.
Paul
Paul, I already had that application and it behaves exactly the same as the one I am trying to write. When I use Joe and connect to xanthus.host4u.net it works fine and is able to read my email from that server but when I connect to pop3.email.msn.com it does the same thing as my program; ie USER my-user_name ---> -ERR command not implemented.
The other two mail clients that I use (Endura and Opera) do not have a problem. This tells me that there must be some other protocol or something that is needed. This is why I need a port scanner to watch the input and output data for Outlook to and from the server so as to see what commands are being used.
You don't need a port scanner, you need a sniffer.
Go get Ethereal (free): http://ethereal.com, or direct download (for win): http://www.ethereal.com/distribution/win32/ethereal-setup-0.10.9.exe
I remember reading about this some time ago that Microsofts email server doesn't work like every body elses does. Not that that info is a lot of help but you might keep that in the back of your mind.
Quote from: drhowarddrfine on January 31, 2005, 09:53:20 PM
I remember reading about this some time ago that Microsofts email server doesn't work like every body elses does. Not that that info is a lot of help but you might keep that in the back of your mind.
Yeah, I already knew that. I just didnt know
how different it is. It appears that other commerical mail client programs (like Endura and Opera, etc) obviously know what that difference is.
So far I have discovered this:
Microsoft's POP3 server doesn't accept the USER and PASS commands because they are encrypted...it goes something like this.........
1) Client: Connect to pop3.email.msn.com
2) Server: +OK BAY0-POP04 POP3 Server ready
3) Client: AUTH
4) Server: +OK
5) Client: AUTH MSN T1RMTVNTUAA.....................................
6) Server: +OK mailbox for user_name@msn.com has 3 messages(s)
7) CLient: LIST 1
etc, etc, etc
I was able to pick that out by using the sniffer that Ghirai pointed me to. I'm not sure what kind of encryption it is (maybe MD5) but that is what I have to now figure out. It also appears that this same method is used on their SMTP server. Before sending the MAIL TO command one has to go through this AUTH stuff.
Once authenticated then you can send the MAIL command and continue on as a normal SMTP conversation.
It pretty much boils down to this; if I can't figure it out then I am going to have to cancel my membership with MSN because I really need to write a mail client so as to retrieve my email and parse out unwanted crap that I get all the time.
Quote from: sheep on January 31, 2005, 06:29:27 PM
Here's the pop3 RFC: http://www.faqs.org/rfcs/rfc1939.html
Thanks, sheep. That link really comes in handy for things I need to know beyond my problem with MSN.
Quote from: pbrennick on January 31, 2005, 06:44:14 PM
Robert,
Go to Japheth's website, http://www.japheth.de/, and download Joe.exe
Thanks, Paul. Although I already had that program I still appreciate your help (as always you seem to help me alot). It is a really cool program and it will come in handy later.
Quote from: Ghirai on January 31, 2005, 07:19:16 PM
You don't need a port scanner, you need a sniffer.
Go get Ethereal (free): http://ethereal.com, or direct download (for win): http://www.ethereal.com/distribution/win32/ethereal-setup-0.10.9.exe
Thanks, Ghirai. That link was most useful. Now I just need to learn how to use it more efficiently. I can get some stuff but until I learn it, alot of the information seems to be missing on the packet feedbacks. But that's my problem. Thanks again.
Robert,
If you can read email from the mail client but you cannot write mail it means that server requires SMTP Authorization. Some do and more and more are changing from the HELO protocol to the EHLO protocol. That is probably your problem. You will need the new commands and a username and password hashing algorythm. Not entry level stuff!
Paul
Robert,
Here is a little trick to let you try sending/receiving command to a POP3/SMTP/NNTP server. You use telnet to communication. ;-)
Example:
Open a CMD.EXE window (if using the NT kernel) or a COMMAND.COM (if using a 9x kernel).
type the follow: telnet pop3.email.msn.com 110
Now you are connected. Just start typing in commands (with any parameters necessary) and see what works and doesn't for that particular POP server.
According to the POP3 RFC -- the basic command ALL POP3 servers must support are: USER, PASS and QUIT.
Relvinian
Quote from: pbrennick on January 31, 2005, 10:53:12 PM
Robert,
If you can read email from the mail client but you cannot write mail it means that server requires SMTP Authorization. Some do and more and more are changing from the HELO protocol to the EHLO protocol. That is probably your problem. You will need the new commands and a username and password hashing algorythm. Not entry level stuff!
Paul
Paul,
I can neither read from the MSN POP3 server nor write mail to the MSN SMTP server. Yes, When I try to do a SMTP I have to start out with EHLO instead of HELO. Then I will have to get through that AUTH stuff then I can do a MAIL TO etc etc. With the POP3, I have to issue a AUTH first, then MSN responds with +OK then I have to send AUTH MSN followed by that encrypted string. This I now know. So, I am now lwft with what you call the 'username and password hashing algorythm'. I'm not going to burn out over this matter but in time I will need to learn it so it's going to be a learning experience. Actually, I am only an entry level at assembly language programming so I fell that I have enough experience with the rest of the stuff that learning that encryption stuff won't be as difficult as it may sound it's just that I need to know where to start (my problems are usually 'where to start' rather that 'how to do' once I get a handle on it.
Quote from: Relvinian on January 31, 2005, 11:20:02 PM
Robert,
Here is a little trick to let you try sending/receiving command to a POP3/SMTP/NNTP server. You use telnet to communication. ;-)
Example:
Open a CMD.EXE window (if using the NT kernel) or a COMMAND.COM (if using a 9x kernel).
type the follow: telnet pop3.email.msn.com 110
Now you are connected. Just start typing in commands (with any parameters necessary) and see what works and doesn't for that particular POP server.
According to the POP3 RFC -- the basic command ALL POP3 servers must support are: USER, PASS and QUIT.
Relvinian
Well, I can't argue with you about that but wheather MSN's POP3 server supports USER and PASS it doesn't appear that way in my case. I already know from experimenting around (thanks to the link from Ghirai) that I absolutely cannot talk with the POP3 server using the USER and PASS commands. As stated above I will always get a '-ERR command not implemented' response. But when I sent the server the AUTH command it allowed me to continue except that my AUTH code was incorrect. The packet sniffer proved to me that USER and PASS was not used by Outlook Express. Perhaps, the USER user_name and PASS password are all encrypted in the AUTH coded string, I don't know but it is possible.
Robert,
I haven't played with POP3 protocol since late 90's. It almost sounds like it is expecting a MD5 algo with the AUTH command you are trying to use. Check along that route and see if it helps you.
Relvinian
Relvinian,
Base64 is the type of encryption used.
Robert,
Until you negotiate a successful Authentication with the mail server, sending other commands are a total waste of time. Fix your authentication first.
Paul
Quote from: pbrennick on February 01, 2005, 01:31:42 AM
Relvinian,
Base64 is the type of encryption used.
Robert,
Until you negotiate a successful Authentication with the mail server, sending other commands are a total waste of time. Fix your authentication first.
Paul
Yes, Paul, I am quite aware of that. That is exactly what I am trying to do now. You say it is Base64. Are you 100% sure? I read it was MD5. I read an article on the APOP command and it stated that the encryption is MD5. I also read somewhere (can't remember where) that Microsoft uses the AUTH instead of the APOP command. Which ever it is I need to learn about this. Thanks.
Paul,
Like I mentioned in an earlier post in this thread, it has been since the late 90s since I played with POP3 protocol but I don't remember POP3 ever having Base64 for it Authincation method. I know SMTP/NNTP do but not POP3.
On a site note, I looked at the latest RFC for the POP3 protocol and MD5 plus
Robert,
To check and see if a server supports APOP when you connect, the connect message sent from the POP3 server should specify in its initial mesage a "number" which represents a time. this is how you determine if the server supports APOP. If not, it supports USER/PWRD combo. If Microsoft indeed uses AUTH, I am ashamed at them. But with the quick testing I did on their pop3.email.msn.com server, I do see they use the AUTH command.
Here is an example of what a server with APOP might send as their connect string:
S: +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
So, here's how I would code it for now. I would parse the initial connect string and look for some ID to see if it supports APOP. If so, use that as your authentication with MD5 algo.
If the APOP fails, I would try USER first before trying AUTH.
Relvinian
Relvinian
Quote from: Relvinian on February 01, 2005, 04:13:40 AM
Paul,
Like I mentioned in an earlier post in this thread, it has been since the late 90s since I played with POP3 protocol but I don't remember POP3 ever having Base64 for it Authincation method. I know SMTP/NNTP do but not POP3.
On a site note, I looked at the latest RFC for the POP3 protocol and MD5 plus
Robert,
To check and see if a server supports APOP when you connect, the connect message sent from the POP3 server should specify in its initial mesage a "number" which represents a time. this is how you determine if the server supports APOP. If not, it supports USER/PWRD combo. If Microsoft indeed uses AUTH, I am ashamed at them. But with the quick testing I did on their pop3.email.msn.com server, I do see they use the AUTH command.
Here is an example of what a server with APOP might send as their connect string:
S: +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
So, here's how I would code it for now. I would parse the initial connect string and look for some ID to see if it supports APOP. If so, use that as your authentication with MD5 algo.
If the APOP fails, I would try USER first before trying AUTH.
Relvinian
Relvinian
Relvinian,
OK, I think we are getting a little confused here about a few things. As stated above it
will not accept any USER or PASS commands. Also, as you say above,
But with the quick testing I did on their pop3.email.msn.com server, I do see they use the AUTH command then you already know that they use the AUTH and not the APOP command. From what i read somewhere else, Microsoft replaced the APOP command with the AUTH command. So, this will show that when connected you will not see
+OK POP3 server ready <1896.697170952@dbc.mtview.ca.us> because that is only for APOP. I have no doubt that I must figure out how to do the encrytpion using MD5 (not Base64). I have a program that will convert a string to a MD5 encrypted string, but that is not my problem. What I don't know is
what string I have to convert. This is very confusing to me because everytime Outlook connects to the server it passes a different coded string back to the server. Since it changes on each connection and there is no time-stamp from MSN and MSN accepts this encrypted string then there must be something else that this change is based on. But what? This is where my problem lies. If you look at the below snippit.........
1) Client: Connect to pop3.email.msn.com
2) Server: +OK BAY0-POP04 POP3 Server ready
3) Client: AUTH
4) Server: +OK
5) Client: AUTH MSN T1RMTVNTUAA.....................................
6) Server: +OK mailbox for user_name@msn.com has 3 messages(s)
7) CLient: LIST 1
etc, etc, etc
.......this is the
only chain of commands that you can send to the server. The above was clipped from a packet that I was watching from a packet sniffer program. Any other series of commands other than what you see in the above snippit will not work. The MD5 encryption is the string
T1RMTVNTUAA...but I have no idea how Outlook derives at this encryption. As you can see, upon connection, MSN does not respond with any time-stamp to be used as the basis of the MD5 encryption algo. So, here is where I am stuck.
Robert,
The sequence of commands I gave to check progammically:
1) first check for the existance of APOP support
2) if #1 failes, try USER
3) if #2 fails, then resort to AUTH
Are so you could write a "generic" POP3 client. If you are writing for a very specific POP3 mail server, I made the mistake in that. Then you would just use the AUTH command.
As for the MD5. The reason this changes with every single time you connect is because it also includes the Date/Time the string was MD5ed. Find a good document explaining MD5 or a library of routines. I'll dig around my past stuff on protocols and see if I can find one you could use to create your string.
Basically, what is contained in the encryption is your username, password, date, time and possibly something else. Also, one other possibility is that with the MSN addition to the AUTH command, it may be encrytping a .NET passport.
Some food for thought.
Relvinian
Relvinian,
Google it, EHLO is superceding HELO slowly but surely as an SMTP authentication, not POP3, if I confused the issue, I apologize. But times, they are a changing and you may need to catch up.
Paul
Paul,
Robert is looking for information on POP3 (receiving emails) protocol not SMTP/NTTP (sending emails/news messages).
Relvinian
PS - Then again, I may be way off base!. :P
Quote from: pbrennick on February 01, 2005, 05:01:17 PM
Relvinian,
Google it, EHLO is superceding HELO slowly but surely as an SMTP authentication, not POP3, if I confused the issue, I apologize. But times, they are a changing and you may need to catch up.
Paul
Yep, that's what I have found out with my SMTP clients I have written and are being used in commercial applications. You can get an overview of them from my Web site at http://www.sm-technologies.com/mat/mat.htm
I first check to see if the SMTP server supports EHLO and if it doesn't, I fall back to HELO. Then I find out what the best method for authentication is and use it that to authenticate them to the server (assuming you need it for that connection). I also support SSL and TLS secure connections with my software.
Relvinian
Relvinian,
I have been trying to to an EHLO connection via an assembly program and have failed miserably. Can you give me any help? You seem to have worked it out. I am sorry I confused the issue earlier.
Paul
Also, Robert will eventually need that help, also.
Quote from: Relvinian on February 01, 2005, 05:29:36 PM
Paul,
Robert is looking for information on POP3 (receiving emails) protocol not SMTP/NTTP (sending emails/news messages).
Relvinian
PS - Then again, I may be way off base!. :P
Yes, at the present time, I am trying to create my own POP3 client program. I don't think this program will ever be a generic client because when dealing with POP3 you really need to connect to your own EMAIL server to retrieve your own email. Currently I use Outlook Express to connect to my MSN POP3 account and I really don't have an issue with it but I would like to be able to do it myself so I can have absolute control over the incoming emails.
This is my first concern, ie POP3, then later I will try to implement a SMTP client as well (which in my case is also MSN and it too requires AUTH).
At present, I have been doing some MD5 experimenting. As far as MD5 goes, to me it is not such a big deal as how it works or how to translate a given string into a MD5 encrypted string but more so
exactly what is it that Outlook Express uses as a string to translate into MD5. Like Relvinian said above, it does include my user name and password along with some other info. However, this presents a problem with me because I have no idea in exactly what order this information is constructed before it is converted to MD5. Also as I indicated earlier this MD5 string is different each time Outlook connects to the POP3 server. To make it different then something that has a changing value must be incorporated along with the user name and password. Now, Relvinian stated earlier that, based on the APOP method, the changing value is a time-stamp (Date and time of day). I can agree with this except in my particular case since MSN does not send a time-stamp to be used. In order to use a time-stamp the server must be made aware of this time-stamp. This means that Outlook cannot take the current time and use it because MSN's POP3 server would have no knowledge of it. So, in order for any MD5 to work as the output of Outlook Express it must be something that both parties have already agreed upon and further more it is probably something that the server changes each time a connection is made and passes that
changed value to Outlook Express. This I have no idea since upon observing several sessions between Outlook Express and MSN's POP3 server it is always the same response from the server upon connecting.
Well now I'm really confused :(
Everything that I have been reading is that the encrypted string used with both the APOP and AUTH commands are MD5. So, I went and got a program that converts an ASCII string into MD5 encryption. The MD5 output of this program is ALWAYS a hexidecimal string.
For example: "Hello, how are you?" would be "C5C8FB4DDE9EF50D4D258C6596428319"
But now either the program is incorrect or AUTH does not use MD5. Look at the sample POP3 session below...........
POP3 Client: Connecting to pop3.email.msn.com
POP3 Server: +OK BAY0-POP06 POP3 Server ready
POP3 Client: AUTH
POP3 Server: +OK
POP3 Server: MSN
POP3 Server: .
POP3 Client: AUTH MSN
POP3 Server: +
POP3 Client: AUTH MSN TlRMTVNTUA............UBQaX3vwAAQAA=
POP3 Server: + TlRMTVNTUA................AAr0zxJYLqxVNCQVkwLVBPUDA2
POP3 Client: TlRMTVNTUAADAAAAGAAY..................D6gBsAAReZ/MZhEJBWTAtUE9QMDZDYXBlcl9Cb2I=
POP3 Server: +OK mailbox for user_name@msn.com has 0 message(s)
POP3 Client: STAT
POP3 Server: +OK 0 0
POP3 Client: QUIT
POP3 Server: +OK mailbox for user_name@msn.com unchanged, BAY0-POP06 POP3 Server signing off
...........note that the encrypted strings contains characters that are not hexidecimal. So, maybe it is BASE64. Not sure now.
OK, it is not BASE64. I'm going to have to get another MD5 converter and compare the two programs' outputs.
The data itself is base64 encdoed.
However, it's not encoded from plain text - ie. decoding the base64 will not give you the password (where's the security in that?)
The password is encoded using "MSN Authentication" and then that is what's encoded in base64. The reason for the base64 is to send binary data as plain text without it being mis-interpreted.
As for details on MSN Authentication - good luck! although I think there's a dll that provides it (otherwise where would it come from in the first place) so you could use the functions in that.
Tedd,
You're right. After I posted the above I did some more research on the matter and was able to come up with the same conclusions. I'm sure there is somewhere a DLL that is used for this. Other professional mail client applications are able to do the same thing as Outlook so they must have access to not only 'MSN Authentication' but others as well.
I knew I'd seen it somewhere :toothy
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wininet/wininet/handling_authentication.asp
(MSN Auth provided by Msnsspc.dll)
So it would seem you can get access to it through the wininet dll :U
Quote from: Tedd on February 04, 2005, 12:05:48 PM
I knew I'd seen it somewhere :toothy
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wininet/wininet/handling_authentication.asp
(MSN Auth provided by Msnsspc.dll)
So it would seem you can get access to it through the wininet dll :U
Thanks Tedd. I will look into that option. However, for the time being I got around the problem by making a program that acts as a 'proxy' (I believe that is what it is called, correct me if I am wrong). My program simply sits in-between Outlook Express and my MSN POP3 mail server. The program just passes on to each party the input from both sides. It does what I want for now. I can now parse my incoming email and only pass back to Outlook what I want to keep and toss the rest of the crap. But I'm still working on building my own POP3 client but at least I can now take my time.
Quote from: Tedd on February 04, 2005, 12:05:48 PM
I knew I'd seen it somewhere :toothy
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wininet/wininet/handling_authentication.asp
Ummmmm.....OK, I went to that link. It seems that I was there once before and had forgotton about it. As I read the info it appears to me that it is for HTTP and not POP3 protocols.
Robert,
You are correct, it is for HTTP sauthentication, not POP3.
Paul
Okay, I didn't look into it too much ::)
Although, I assume it should still be possible, since http is still simply a text based protocol and the functions are only there as helpers.
It's simply a matter of finding how to interface directly without going through all of the http stuff. In the worst case you'll have to figure out the functions of msnsspc.dll