Uploading multiple water temperature files to CDX/WQX Exchange Network

Upload tool for submitting multiple files at once to the WQX

We have more than 100 xml files of water temperature to upload to the WQX water quality exchange, which is an Axis endpoint located here:

Water Quality Exchange (WQX)

Node 2.1 Test – https://testngn.epacdxnode.net/ngn-enws20/services/NetworkNode2ServiceConditionalMTOM
Node 2.1 Production – https://cdxnodengn.epa.gov/ngn-enws20/services/NetworkNode2ServiceConditionalMTOM
Node 1.1 Test – https://testngn.epacdxnode.net/cdx-enws10/services/NetworkNodePortType_V10
Node 1.1 Production – https://cdxnodengn.epa.gov/cdx-enws10/services/NetworkNodePortType_V10

Getting the account(s) setup

WQXWeb account

First I had to get my WQX Web account setup.  I contacted the STORET folks who setup my account by following these instructions: http://www.epa.gov/storet/wqx/wqxweb.html.  I emailed their helpdesk with the information they asked for and they setup my account. Then I could login to http://cdx.epa.gov and view our organization's data as well as manually upload individual files.

Rather than upload the files individually through the web interface, I wanted to upload the files in bulk.  On the getting started with WQX page (http://www.epa.gov/storet/wqx/wqx.html) they suggest using the Windsor Node Client Lite 2 software (http://www.windsorsolutions.biz/nodeclient/disclaimer.html) so I downloaded it and followed the instructions on the getting started page for getting the endpoints configured.  But I ran into a problem.

NAAS Account

As it turns out, the username/password that STORET gives you is not a NAAS account.  You need to have a NAAS account in order to submit data using the endpoints above, which includes submitting files using the Node Client Lite 2 software.

To setup a NAAS account, I needed to email nodehelpdesk@epacdx.net and request a NAAS account that is linked to my STORET account and provide them with my STORET account id and organization id.  They contacted the STORET helpdesk and asked for their approval to set me up.  Once I was approved, they sent me my new, shiny NAAS account.

Using the new NAAS credentials to login with the Node Client Lite 2 software, I was able to submit a file to the test node.  Once that was successful, I needed to contact the nodehelpdesk@epacdx.net guys again and give them the successfully submitted transaction id so that they could verify that everything about my file was good and then they enabled me for submitting to the production node.

Now I thought I was all set!  I selected a bunch of files through the Node Client Lite user interface but received an error that the endpoint expected one file but received many.  I asked the helpdesk about it and they informed me that I could only send one file at a time.  Bummer!  So I decided to write my own client that would upload my files.  We want to automate this whole process in the future anyway, so I thought I'd just whip up a connector.

Writing WQXUpload tool

These days, I'm writing in c# using Visual Studio 2012.  So I fired it up and created a new project.  Then I went to PROJECT -> add service reference, clicked "Advanced" button and then "Add Web Reference".  I pasted the URL for the endpoint I'm using (https://testngn.epacdxnode.net/cdx-enws10/services/NetworkNodePortType_V10 I'd definitely start with the test url) and then let it build my SOAP connector class automagically.

Then here is my code for the uploader.  Enjoy!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using ConsoleApplication1.gov.epa.cdxnodengn;
using ConsoleApplication1.gov.epa.cdx;
using System.IO;

namespace ConsoleApplication1
{
    class WQXUploader
    {
        private static string defaultFilePath = "."; //all .xml files in this directory will be uploaded.

        private static NetworkNode node;
        private static string token;
        private static int numFilesUploaded = 0;


        static int Main(string[] args)
        {

            if (args.Length < 2 )
            {
                log("Syntax: WQXUpload   [path] (default path is current directory)");
                waitForKeypress();
                return 1;
            }

            //setup our passed in credentials.
            string username = args[0];
            string password = args[1];

            //if they call us with a path then set it here.
            string filePath = defaultFilePath;
            if (args.Length == 3)
            {
                filePath = args[2];
            }

            //done with args now get rolling...
            log("Starting WQX Uploader.");
            Inititialize();
            Authenticate(username, password);

            //which files to upload?
            List filesToUpload = getFilesToUpload(filePath);

            //submit each file.
            foreach (var file in filesToUpload)
            {
                //log("Uploading file: " + file);
                submitFile(token, file, filePath);
                numFilesUploaded++;
            }

            Cleanup();
            log("WQX upload is finished. We uploaded "+ numFilesUploaded + " files." );
            waitForKeypress();

            return 0;
        }

        /**
         * Gets a list of files from the filepath that we will upload
         * @return List files to upload
         */ 
        private static List getFilesToUpload(string filePath)
        {
            log("Using path: " + filePath);

            List files = new List();

            DirectoryInfo di = new DirectoryInfo(filePath);
            var directories = di.GetFiles("*.xml", SearchOption.TopDirectoryOnly);

            foreach (FileInfo file in directories)
            {
                files.Add(file.Name);
            }

            return files;
        }

        /*
         * Wait for a keypress -- just to keep the console open.
         */ 
        private static void waitForKeypress()
        {
            Console.WriteLine("Press ESC to exit");
            do
            {
                while (!Console.KeyAvailable)
                {
                    // Do something
                }
            } while (Console.ReadKey(true).Key != ConsoleKey.Escape);
        }

        /**
         * Submits file to WQX
         * @return transactionId of the submission.
         */ 
        private static string submitFile(string token, string fileToUpload, string filePath)
        {
            NodeDocument doc = new NodeDocument();

            byte[] buffer = File.ReadAllBytes(filePath + "\\" + fileToUpload);

            doc.name = fileToUpload;
            doc.type = "xml";
            doc.content = buffer; 

            NodeDocument[] docToSend = { doc };

            String transactionId = node.Submit(token, "", "WQX", docToSend);

            log(fileToUpload + " " + transactionId);
            //logStatus(transactionId);

            return transactionId;

        }

        /*
         * Outputs the status of the transaction to the log.
         */ 
        private static void logStatus(string transactionId)
        {
            log(" Current status: " + node.GetStatus(token, transactionId));
        }

        /*
         * Writes the message to the log
         */ 
        private static void log(string msg)
        {
            Console.WriteLine(msg);
        }

        /*
         * Initializes the NetworkNode connection
         */ 
        private static void Inititialize()
        {
            node = new NetworkNode();
        }

        /**
         * Performs any cleanup necessary
         */ 
        private static void Cleanup()
        {
            node.Dispose();
        }

        /*
         * Connects and authenticates us with the endpoint, fetching our token necessary for other functions.
         * @return string token
         */ 
        private static string Authenticate(String username, String password)
        {
            log("Connecting and authenticating...");
            token = node.Authenticate(username, password, "password");
            log("Success! Security token: " + token);
            return token;
        }
    }
}