Last week I had a big headache trying to import a 165MB XML file. I couldn’t even open it with any browse and reading it with ColdFusion gave me 500 errors, exceeded heap size, JRun closed connection, etc…

The solution I found was to read the file line-by-line using Java and creating blocks of XML code that allowed me to parse into objects and process them individually. Remembering, we use CF7 yet.

So I wrote this piece of code where I set my start tag and it will grab everything between my start and end tag, and assemble a XML object from there. I know the CF Gurus would suggest a much better solution, but that’s what i could accomplish with my humble knowledge and give a quick solution for the problem.

Imagine you received a file from a book editor with thousands of books. The format is basically the following:
{code type=xml}<?xml version=”1.0″ encoding=”UTF-8″?>
<books>
<book>
<bookid>1234</bookid>
<title>My Best Book</title>
<author>John Doe</author>
<price>49.99</price>
</book>
…. …. ….
</books>{/code}

Let’s set our start tag as “book”, because we want to process the content of each book.  Here is the code:

{code type=coldfusion}<cfscript>
filePath = “c:\upload\books.xml”;
xmlHeader = ‘<?xml version=”1.0″ encoding=”UTF-8″?>’;
rootStart = “books”;
startTagText = “book”;
myXmlText = “”;
// instantiate the java objects
hFile = CreateObject(“Java”,”java.io.FileReader”).init(filePath);
hFile = createObject(“java”,”java.io.BufferedReader”).init(hFile);
startTagFound = false;
endOfFile = false;
while (not endOfFile) {
// read a line
line = hFile.readLine();
if (len(line)) {
// if the line exists, call function to mount it, passing the start tag
result = mountMyLine(line, startTagText);
// if the result does not contain the start tag, discard it, skip to next line
if (not result.hasStartTag and not startTagFound) continue;
// if result has the start tag and it is the first time, save the text
if (not startTagFound and result.hasStartTag) {
startTagFound = true;
myXmlText = myXmlText & result.text;
// and if it is not the first time and result does not have the end tag, save the text
} else if (startTagFound and not result.hasEndTag) {
myXmlText = myXmlText & result.text;
// or if the result has end tag, save the text and parse as xml object
} else if (result.hasEndTag) {
myXmlText = myXmlText & result.text;
myDoc = xmlParse(processMyText(myXmlText, outFilePath, xmlHeader, rootStart, startTagText));
book = myDoc.xmlRoot.book;
startTagFound = false;
myXmltext = “”;
// do something here with the object, insert into a table, etc…
}
}
}
</cfscript>{/code}

Here is the function mountMyLine():

{code type=coldfusion}<cfscript>
function mountMyLine(lineIn, tagStart) {
var start = 1;
var end = 0;
var pos = 0;
var line = “”;
var startTag = “<” & tagStart & “>”;
var endTag = “</” & tagStart & “>”;
var result = structNew();
result.hasStartTag = false;
result.hasEndTag = false;
line = replace(lineIn, chr(13), “”, “all”);
line = replace(line, chr(10), “”, “all”);
line = trim(line);
end = len(line);
// find start tag
pos = findNoCase(startTag, line);
if ( pos gt 0) {
start = pos + len(startTag);
result.hasStartTag = true;
}
// find end tag
pos = findNoCase(endTag, line);
if (pos gt 0) {
end = pos – start ;
result.hasEndTag = true;
}
result.text = mid(line, start, end);
return result;
}
</cfscript>{/code}

Here is the function processMyText():

{code type=coldfusion}<cfscript>
function processMyText(text,xmlHeader,rootStartText,startTagText) {
var rootStart = “<” & arguments.rootStartText & “>”;
var rootEnd = “</” & arguments.rootStartText & “>”;
var startTag = “<” & arguments.startTagtext & “>”;
var endTag = “</” & arguments.startTagText & “>”;
var mytext = arguments.xmlHeader & rootStart & startTag & arguments.text & endTag & rootEnd;
}
</cfscript>{/code}

Well, it solved my problem, although I know someone could write a better code using regular expressions to retrieve the content. I’d be glad if someone could share other better solutions or point my mistakes on this one.