j-j.co.za

As in the previous post, our initial NMAP scan produced an XML file over 600mb in size. To finish the scans we split the remaining ip ranges into more manageable chunks and ended up with another 20+ xml files of around 50mb each.

Running all of these through Exult XML to get a single consolidated access database was a bit problematic. The tool didn’t have the functionality to add additional scans to our original database, so all of the XML files had to be selected together and run through the tool to produce a single database. The conversion ran for 24hrs without completing so we had to come up with a better plan. Initially we considered running the conversion on a more powerful machine with much faster disk, but when trying to install the tool discovered the license key wouldn’t work.  It used some (undisclosed) technique to ensure single install only. An email off to the developers and they sent us a new key (about 6 hours later – thanks to time differences, not any delay on their part). In the meantime plan B was in place.

Looking through the raw XML files I saw that the vast majority of the IP’s were non responsive and those few lines indicating this were taking up an awful lot of space when looked at collectively. A quick search and replace to remove these lines and the XML files were reduced to about 1/20th of their original size. With the new reduced file sizes Exult happily produced our Access database in less than half an hour and we were ready to continue with the exercise. The old sayings about better planning and new strategies certainly applied here.

Using access we were then able to produce great summary reports to pull out details on top active ports, numbers of machines in each class, active IP ranges etc etc. Next step choose our samples for more detailed scanning.

Over the last few days we ran a really large discovery scan on a client’s network. The scan was discontinued part of the way through and at that stage had produced a 650Mb .xml file. Smaller files are easy to load into Excel or Firefox to view and work with. With this much data we needed a more workable solution.

The first though was to import it into MS Access 2007. Access has built in import filters, easy enough we though. Two error lines in a table and no data brought an end to that hopeful idea. A little bit of googling found as a tool that promised to do pain free importing of XML into access, building the table structures on the fly and automatically creating the necessary table links.

The Exult XML converter from novixys.com was a lifesaver.We downloaded the trial, tested it on a scan of the local class C and it worked wonders, creating all the required tables and links. $105 later on the credit card and the full version was purchased.

Using it on out full scan file was a little trickier. Since we aborted the scan the .xml file hadn’t been properly completed. Exult didn’t like the incomplete file so threw out an error message and refused to build the access database.  A comparison of the short test scan and the full scan revealed the missing XML tags. We copied and pasted these over to the full file, updating the scan information manually, and saved the file. Running Exult again took a while (over an hour) but produced the required file. Perfect solution.

Thanks Novixsys.

We are scanning the rest of the network as we speak. Not sure how we will import the balance of the .xml files into the same database. Hopefully Exult can do that for us too. Will find out soon enough